[I 2022-11-02 09:20:38] ...ytorch/bootstrap/run.py.73: Saving environment info
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75: options: 
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:   dataset:
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     aux_kw_path:
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       test: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_recipe1m_13m_orig_test_titles_kw.json
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       train: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_recipe1m_13m_train_titles_kw.json
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       val: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_recipe1m_13m_orig_val_titles_kw.json
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     batch_sampler: random
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     batch_size: 100
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     database: lmdb
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     dir: /gpfsdswork/dataset/Recipe1M+/images_recipe1M+
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     eval_split: test
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     freq_mismatch: 0.0
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     image_from: database
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     import: recipe1m.datasets.factory
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     kw_path:
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       test: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_recipe1m_13m_orig_test_ingr_kw.json
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       train: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_recipe1m_13m_train_ingr_kw.json
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       val: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_recipe1m_13m_orig_val_ingr_kw.json
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     name: recipe1m_13m
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     nb_classes: 1048
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     nb_threads: 4
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     orig_ids_eval: True
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     path_ids: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/recipe1m_13m/original_ids.json
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     path_image_json: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/recipe1m_13m/layer2+.json
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     path_text_json: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/text/tokenized_layer1.json
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     randkw_p: 0.3
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     randkw_p_aux: 0.5
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     tokenized_raw_text: True
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     train_split: None
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     use_vcs: True
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     vocab_path: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/text/vocab_all.txt
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:   engine:
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     name: logger
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     nb_epochs: 60
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     print_freq: 10
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     saving_criteria: ['eval_epoch.metric.med_im2recipe_mean:min', 'eval_epoch.metric.recall_at_1_im2recipe_mean:max']
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:   exp:
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     dir: /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_recipe1m_13m_orig_ids
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     resume: best_eval_epoch.metric.recall_at_1_im2recipe_mean
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:   misc:
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     cuda: True
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     data_parrallel: True
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     logs_name: None
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     overrite: False
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     seed: 1338
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:   model:
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     criterion:
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:         tri_cross_encoder: False
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       itm_loss_weight: 1
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       keep_background: False
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       retrieval_strategy:
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:         aggregation: valid
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:         get_tokens: True
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:         margin: 0.05
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:         margin_params:
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:           increment: 0.005
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:           increment_margin: True
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:           max_margin: 0.3
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:         name: vlp_triplet
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:         nb_samples: 9999
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:         sampling: max_negative
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:         substrategy: ['IRR', 'RII']
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:         substrategy_weights: [1.0, 1.0]
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     import: recipe1m.models.factory
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     metric:
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       k_test: 10
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       nb_bags: 10
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       nb_matchs_per_bag: 1000
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       retrieval_dir: /gpfsscratch/rech/dyf/ugz83ue/tmp/recipe1m
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       trijoint: True
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     name: trijoint
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     network:
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       activations: ['tanh', 'normalize']
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       aux_kwords: True
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       aux_kwords_encoder: True
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       avg_concat: True
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       bert_config: /linkhome/rech/genisi01/ugz83ue/tfood/recipe1m/models/networks/recipe_networks/config_bert_albef.json
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       cat_pos: 0
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       class_attention: False
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       class_attention_fus: False
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       cls_token: False
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       cls_token_fus: False
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       cross_decoder_image: True
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       cross_decoder_recipe: True
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:         class_attention: False
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:         cls_token: False
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:         cross_attention: False
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:         cross_decoder: True
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:         double_decoder: False
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:         get_tokens: False
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:         load_backbones: False
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:         n_heads: 4
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:         n_layers: 4
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:         query_img: False
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       cross_transformer: False
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       dim_emb: 1024
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       dim_image_out: 2048
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       dim_ingr_out: 300
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       dim_instr_in: 1024
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       dim_instr_out: 1024
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       freeze_im: False
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       freeze_rec: False
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       get_tokens: True
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       get_tokens_cross_decoder_recipe: True
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       hidden_size: 512
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       image_backbone_name: clip
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       image_res: 224
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       keep_embedding: True
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       kwords_same_level: False
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       last_cat_aux: True
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       n_heads: 4
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       n_heads_cross: 4
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       n_heads_cross_image: 4
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       n_heads_single: 4
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       n_layers: 2
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       n_layers_cross: 2
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       n_layers_cross_image: 1
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       n_layers_single: 2
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       num_hidden_layers_kw: 2
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       old: True
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       path_ingrs: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/text/vocab.pkl
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       path_vocab: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/text/vocab_all.txt
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       recipe_encoder: tfoodvicha-transformer
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       text_encoder: bert-base-uncased
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       vision_width: 768
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       vit_all_tokens: True
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       with_ingrs: True
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       with_instrs: True
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:       with_titles: True
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     with_classif: False
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:   optimizer:
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     batch_size_factor: None
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     clip_grad: 8.0
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     import: recipe1m.optimizers.factory
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     lr: 1e-05
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     lr_img: 1e-06
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     name: trijoint_fixed_fine_tune
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:     switch_epoch: 5
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:   path_opts: /linkhome/rech/genisi01/ugz83ue/tfood/recipe1m/options/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_recipe1m_13m.yaml
[S 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.75:   view: ['logs:train_epoch.loss', 'logs:train_epoch.bad_pairs', 'logs:eval_epoch.metric.med_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_1_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_5_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_10_im2recipe_mean', 'logs:eval_epoch.metric.med_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_1_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_5_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_10_recipe2im_mean', 'logs:optimizer.is_optimizer_recipe&image', 'logs:optimizer.total_norm']
[I 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.76: posix.uname_result(sysname='Linux', nodename='jean-zay-ia822', release='4.18.0-305.57.1.el8_4.x86_64', version='#1 SMP Tue Jul 5 17:55:55 EDT 2022', machine='x86_64')
[I 2022-11-02 09:20:43] ...ytorch/bootstrap/run.py.80: Available GPUs: [0]
[I 2022-11-02 09:20:43] ...strap/engines/factory.py.9: Creating engine...
[I 2022-11-02 09:20:43] ...trap/datasets/factory.py.8: Creating dataset...
[I 2022-11-02 09:22:32] ...tasets/recipe1m_13m.py.290: Load VCs...
[I 2022-11-02 09:22:33] ...tasets/recipe1m_13m.py.303: randkw_p... None
[I 2022-11-02 09:22:33] ...rap/datasets/factory.py.21: Evaluation will take place on testset (152388 items)
[I 2022-11-02 09:22:33] ...strap/models/factory.py.21: Creating model...
[I 2022-11-02 09:22:37] ...e_networks/networks.py.203: Loading CLIP Visual model...
[I 2022-11-02 09:22:40] ...e_networks/networks.py.145: cuda
[I 2022-11-02 09:22:42] ...networks/tfood_vicha.py.48: Load VCs encoder...
[I 2022-11-02 09:22:43] ...els/metrics/trijoint.py.46: self.save_ids False
[I 2022-11-02 09:22:43] ...strap/models/factory.py.44: Enabling CUDA mode...
[I 2022-11-02 09:22:49] ...etworks/tfood_vicha.py.183: training aux kw modules
[I 2022-11-02 09:22:49] ...etworks/tfood_vicha.py.183: training aux kw modules
[I 2022-11-02 09:22:49] ...otstrap/views/factory.py.8: Creating views...
[I 2022-11-02 09:22:52] ...strap/engines/engine.py.98: Loading best_eval_epoch.metric.recall_at_1_im2recipe_mean checkpoint
[I 2022-11-02 09:22:52] ...trap/engines/engine.py.440: Loading model...
[I 2022-11-02 09:22:58] ...trap/engines/engine.py.453: Loading engine...
[I 2022-11-02 09:22:58] ...trap/engines/engine.py.108: Launching evaluation procedures
[I 2022-11-02 09:22:58] ...trap/engines/engine.py.268: Evaluating model on testset for epoch 59
[I 2022-11-02 09:22:58] ...tasets/recipe1m_13m.py.468: Dataset will be sampled with "random" batch_sampler.
[I 2022-11-02 09:23:07] ...trap/engines/engine.py.321: eval: epoch 59 | batch 0/1522
[I 2022-11-02 09:23:07] ...trap/engines/engine.py.325:       elapsed: 0:00:09 | left: 3:54:51
[I 2022-11-02 09:23:07] ...trap/engines/engine.py.326:       process: 9.25868 | load: 6.84397
[I 2022-11-02 09:23:20] ...trap/engines/engine.py.321: eval: epoch 59 | batch 10/1522
[I 2022-11-02 09:23:20] ...trap/engines/engine.py.325:       elapsed: 0:00:22 | left: 0:54:38
[I 2022-11-02 09:23:20] ...trap/engines/engine.py.326:       process: 1.16830 | load: 0.00007
[I 2022-11-02 09:23:36] ...trap/engines/engine.py.321: eval: epoch 59 | batch 20/1522
[I 2022-11-02 09:23:36] ...trap/engines/engine.py.325:       elapsed: 0:00:38 | left: 0:42:43
[I 2022-11-02 09:23:36] ...trap/engines/engine.py.326:       process: 2.22947 | load: 1.02655
[I 2022-11-02 09:23:50] ...trap/engines/engine.py.321: eval: epoch 59 | batch 30/1522
[I 2022-11-02 09:23:50] ...trap/engines/engine.py.325:       elapsed: 0:00:51 | left: 0:34:20
[I 2022-11-02 09:23:50] ...trap/engines/engine.py.326:       process: 1.17745 | load: 0.00013
[I 2022-11-02 09:24:04] ...trap/engines/engine.py.321: eval: epoch 59 | batch 40/1522
[I 2022-11-02 09:24:04] ...trap/engines/engine.py.325:       elapsed: 0:01:06 | left: 0:40:18
[I 2022-11-02 09:24:04] ...trap/engines/engine.py.326:       process: 3.03655 | load: 1.78690
[I 2022-11-02 09:24:19] ...trap/engines/engine.py.321: eval: epoch 59 | batch 50/1522
[I 2022-11-02 09:24:19] ...trap/engines/engine.py.325:       elapsed: 0:01:20 | left: 0:35:57
[I 2022-11-02 09:24:19] ...trap/engines/engine.py.326:       process: 1.18461 | load: 0.00012
[I 2022-11-02 09:24:34] ...trap/engines/engine.py.321: eval: epoch 59 | batch 60/1522
[I 2022-11-02 09:24:34] ...trap/engines/engine.py.325:       elapsed: 0:01:35 | left: 0:37:24
[I 2022-11-02 09:24:34] ...trap/engines/engine.py.326:       process: 2.29772 | load: 1.11799
[I 2022-11-02 09:24:48] ...trap/engines/engine.py.321: eval: epoch 59 | batch 70/1522
[I 2022-11-02 09:24:48] ...trap/engines/engine.py.325:       elapsed: 0:01:50 | left: 0:34:36
[I 2022-11-02 09:24:48] ...trap/engines/engine.py.326:       process: 1.17983 | load: 0.00013
[I 2022-11-02 09:25:02] ...trap/engines/engine.py.321: eval: epoch 59 | batch 80/1522
[I 2022-11-02 09:25:02] ...trap/engines/engine.py.325:       elapsed: 0:02:04 | left: 0:35:35
[I 2022-11-02 09:25:02] ...trap/engines/engine.py.326:       process: 2.10342 | load: 0.92885
[I 2022-11-02 09:25:16] ...trap/engines/engine.py.321: eval: epoch 59 | batch 90/1522
[I 2022-11-02 09:25:16] ...trap/engines/engine.py.325:       elapsed: 0:02:18 | left: 0:34:05
[I 2022-11-02 09:25:16] ...trap/engines/engine.py.326:       process: 1.18442 | load: 0.00013
[I 2022-11-02 09:25:36] ...trap/engines/engine.py.321: eval: epoch 59 | batch 100/1522
[I 2022-11-02 09:25:36] ...trap/engines/engine.py.325:       elapsed: 0:02:37 | left: 0:42:04
[I 2022-11-02 09:25:36] ...trap/engines/engine.py.326:       process: 2.56576 | load: 1.47771
[I 2022-11-02 09:25:48] ...trap/engines/engine.py.321: eval: epoch 59 | batch 110/1522
[I 2022-11-02 09:25:48] ...trap/engines/engine.py.325:       elapsed: 0:02:50 | left: 0:31:07
[I 2022-11-02 09:25:48] ...trap/engines/engine.py.326:       process: 1.07950 | load: 0.00012
[I 2022-11-02 09:26:03] ...trap/engines/engine.py.321: eval: epoch 59 | batch 120/1522
[I 2022-11-02 09:26:03] ...trap/engines/engine.py.325:       elapsed: 0:03:05 | left: 0:37:08
[I 2022-11-02 09:26:03] ...trap/engines/engine.py.326:       process: 2.68265 | load: 1.60540
[I 2022-11-02 09:26:18] ...trap/engines/engine.py.321: eval: epoch 59 | batch 130/1522
[I 2022-11-02 09:26:18] ...trap/engines/engine.py.325:       elapsed: 0:03:20 | left: 0:34:05
[I 2022-11-02 09:26:18] ...trap/engines/engine.py.326:       process: 1.08879 | load: 0.00014
[I 2022-11-02 09:26:33] ...trap/engines/engine.py.321: eval: epoch 59 | batch 140/1522
[I 2022-11-02 09:26:33] ...trap/engines/engine.py.325:       elapsed: 0:03:34 | left: 0:34:46
[I 2022-11-02 09:26:33] ...trap/engines/engine.py.326:       process: 2.29066 | load: 1.20938
[I 2022-11-02 09:26:46] ...trap/engines/engine.py.321: eval: epoch 59 | batch 150/1522
[I 2022-11-02 09:26:46] ...trap/engines/engine.py.325:       elapsed: 0:03:48 | left: 0:31:51
[I 2022-11-02 09:26:46] ...trap/engines/engine.py.326:       process: 1.08031 | load: 0.00013
[I 2022-11-02 09:27:02] ...trap/engines/engine.py.321: eval: epoch 59 | batch 160/1522
[I 2022-11-02 09:27:02] ...trap/engines/engine.py.325:       elapsed: 0:04:03 | left: 0:36:39
[I 2022-11-02 09:27:02] ...trap/engines/engine.py.326:       process: 2.74270 | load: 1.64719
[I 2022-11-02 09:27:15] ...trap/engines/engine.py.321: eval: epoch 59 | batch 170/1522
[I 2022-11-02 09:27:15] ...trap/engines/engine.py.325:       elapsed: 0:04:17 | left: 0:30:32
[I 2022-11-02 09:27:15] ...trap/engines/engine.py.326:       process: 1.08314 | load: 0.00013
[I 2022-11-02 09:27:32] ...trap/engines/engine.py.321: eval: epoch 59 | batch 180/1522
[I 2022-11-02 09:27:32] ...trap/engines/engine.py.325:       elapsed: 0:04:33 | left: 0:36:51
[I 2022-11-02 09:27:32] ...trap/engines/engine.py.326:       process: 2.65509 | load: 1.57307
[I 2022-11-02 09:27:44] ...trap/engines/engine.py.321: eval: epoch 59 | batch 190/1522
[I 2022-11-02 09:27:44] ...trap/engines/engine.py.325:       elapsed: 0:04:46 | left: 0:28:36
[I 2022-11-02 09:27:44] ...trap/engines/engine.py.326:       process: 1.08070 | load: 0.00013
[I 2022-11-02 09:28:01] ...trap/engines/engine.py.321: eval: epoch 59 | batch 200/1522
[I 2022-11-02 09:28:01] ...trap/engines/engine.py.325:       elapsed: 0:05:03 | left: 0:37:35
[I 2022-11-02 09:28:01] ...trap/engines/engine.py.326:       process: 1.78687 | load: 0.70172
[I 2022-11-02 09:28:16] ...trap/engines/engine.py.321: eval: epoch 59 | batch 210/1522
[I 2022-11-02 09:28:16] ...trap/engines/engine.py.325:       elapsed: 0:05:18 | left: 0:32:56
[I 2022-11-02 09:28:16] ...trap/engines/engine.py.326:       process: 1.04400 | load: 0.00013
[I 2022-11-02 09:28:33] ...trap/engines/engine.py.321: eval: epoch 59 | batch 220/1522
[I 2022-11-02 09:28:33] ...trap/engines/engine.py.325:       elapsed: 0:05:34 | left: 0:38:06
[I 2022-11-02 09:28:33] ...trap/engines/engine.py.326:       process: 3.08099 | load: 1.99744
[I 2022-11-02 09:28:48] ...trap/engines/engine.py.321: eval: epoch 59 | batch 230/1522
[I 2022-11-02 09:28:48] ...trap/engines/engine.py.325:       elapsed: 0:05:50 | left: 0:34:36
[I 2022-11-02 09:28:48] ...trap/engines/engine.py.326:       process: 1.08282 | load: 0.00012
[I 2022-11-02 09:29:03] ...trap/engines/engine.py.321: eval: epoch 59 | batch 240/1522
[I 2022-11-02 09:29:03] ...trap/engines/engine.py.325:       elapsed: 0:06:04 | left: 0:33:56
[I 2022-11-02 09:29:03] ...trap/engines/engine.py.326:       process: 2.81365 | load: 1.74013
[I 2022-11-02 09:29:17] ...trap/engines/engine.py.321: eval: epoch 59 | batch 250/1522
[I 2022-11-02 09:29:17] ...trap/engines/engine.py.325:       elapsed: 0:06:19 | left: 0:30:34
[I 2022-11-02 09:29:17] ...trap/engines/engine.py.326:       process: 1.08172 | load: 0.00013
[I 2022-11-02 09:29:33] ...trap/engines/engine.py.321: eval: epoch 59 | batch 260/1522
[I 2022-11-02 09:29:33] ...trap/engines/engine.py.325:       elapsed: 0:06:34 | left: 0:34:17
[I 2022-11-02 09:29:33] ...trap/engines/engine.py.326:       process: 2.75346 | load: 1.67739
[I 2022-11-02 09:29:46] ...trap/engines/engine.py.321: eval: epoch 59 | batch 270/1522
[I 2022-11-02 09:29:46] ...trap/engines/engine.py.325:       elapsed: 0:06:48 | left: 0:28:38
[I 2022-11-02 09:29:46] ...trap/engines/engine.py.326:       process: 1.08331 | load: 0.00015
[I 2022-11-02 09:30:02] ...trap/engines/engine.py.321: eval: epoch 59 | batch 280/1522
[I 2022-11-02 09:30:02] ...trap/engines/engine.py.325:       elapsed: 0:07:04 | left: 0:33:58
[I 2022-11-02 09:30:02] ...trap/engines/engine.py.326:       process: 2.38833 | load: 1.29934
[I 2022-11-02 09:30:16] ...trap/engines/engine.py.321: eval: epoch 59 | batch 290/1522
[I 2022-11-02 09:30:16] ...trap/engines/engine.py.325:       elapsed: 0:07:18 | left: 0:28:53
[I 2022-11-02 09:30:16] ...trap/engines/engine.py.326:       process: 1.06571 | load: 0.00015
[I 2022-11-02 09:30:32] ...trap/engines/engine.py.321: eval: epoch 59 | batch 300/1522
[I 2022-11-02 09:30:32] ...trap/engines/engine.py.325:       elapsed: 0:07:34 | left: 0:31:37
[I 2022-11-02 09:30:32] ...trap/engines/engine.py.326:       process: 1.99573 | load: 0.91148
[I 2022-11-02 09:30:47] ...trap/engines/engine.py.321: eval: epoch 59 | batch 310/1522
[I 2022-11-02 09:30:47] ...trap/engines/engine.py.325:       elapsed: 0:07:48 | left: 0:28:05
[I 2022-11-02 09:30:47] ...trap/engines/engine.py.326:       process: 1.09967 | load: 0.00015
[I 2022-11-02 09:31:03] ...trap/engines/engine.py.321: eval: epoch 59 | batch 320/1522
[I 2022-11-02 09:31:03] ...trap/engines/engine.py.325:       elapsed: 0:08:04 | left: 0:30:42
[I 2022-11-02 09:31:03] ...trap/engines/engine.py.326:       process: 1.08364 | load: 0.00016
[I 2022-11-02 09:31:16] ...trap/engines/engine.py.321: eval: epoch 59 | batch 330/1522
[I 2022-11-02 09:31:16] ...trap/engines/engine.py.325:       elapsed: 0:08:18 | left: 0:26:39
[I 2022-11-02 09:31:16] ...trap/engines/engine.py.326:       process: 1.08037 | load: 0.00016
[I 2022-11-02 09:31:33] ...trap/engines/engine.py.321: eval: epoch 59 | batch 340/1522
[I 2022-11-02 09:31:33] ...trap/engines/engine.py.325:       elapsed: 0:08:35 | left: 0:31:07
[I 2022-11-02 09:31:33] ...trap/engines/engine.py.326:       process: 1.54679 | load: 0.46236
[I 2022-11-02 09:31:48] ...trap/engines/engine.py.321: eval: epoch 59 | batch 350/1522
[I 2022-11-02 09:31:48] ...trap/engines/engine.py.325:       elapsed: 0:08:50 | left: 0:29:36
[I 2022-11-02 09:31:48] ...trap/engines/engine.py.326:       process: 1.08496 | load: 0.00015
[I 2022-11-02 09:32:03] ...trap/engines/engine.py.321: eval: epoch 59 | batch 360/1522
[I 2022-11-02 09:32:03] ...trap/engines/engine.py.325:       elapsed: 0:09:05 | left: 0:28:06
[I 2022-11-02 09:32:03] ...trap/engines/engine.py.326:       process: 1.19440 | load: 0.11349
[I 2022-11-02 09:32:18] ...trap/engines/engine.py.321: eval: epoch 59 | batch 370/1522
[I 2022-11-02 09:32:18] ...trap/engines/engine.py.325:       elapsed: 0:09:20 | left: 0:28:28
[I 2022-11-02 09:32:18] ...trap/engines/engine.py.326:       process: 1.08459 | load: 0.00007
[I 2022-11-02 09:32:35] ...trap/engines/engine.py.321: eval: epoch 59 | batch 380/1522
[I 2022-11-02 09:32:35] ...trap/engines/engine.py.325:       elapsed: 0:09:36 | left: 0:32:03
[I 2022-11-02 09:32:35] ...trap/engines/engine.py.326:       process: 2.37772 | load: 1.30043
[I 2022-11-02 09:32:50] ...trap/engines/engine.py.321: eval: epoch 59 | batch 390/1522
[I 2022-11-02 09:32:50] ...trap/engines/engine.py.325:       elapsed: 0:09:52 | left: 0:29:15
[I 2022-11-02 09:32:50] ...trap/engines/engine.py.326:       process: 1.08521 | load: 0.00007
[I 2022-11-02 09:33:06] ...trap/engines/engine.py.321: eval: epoch 59 | batch 400/1522
[I 2022-11-02 09:33:06] ...trap/engines/engine.py.325:       elapsed: 0:10:07 | left: 0:28:19
[I 2022-11-02 09:33:06] ...trap/engines/engine.py.326:       process: 1.80188 | load: 0.70639
[I 2022-11-02 09:33:22] ...trap/engines/engine.py.321: eval: epoch 59 | batch 410/1522
[I 2022-11-02 09:33:22] ...trap/engines/engine.py.325:       elapsed: 0:10:24 | left: 0:29:30
[I 2022-11-02 09:33:22] ...trap/engines/engine.py.326:       process: 1.08127 | load: 0.00007
[I 2022-11-02 09:33:38] ...trap/engines/engine.py.321: eval: epoch 59 | batch 420/1522
[I 2022-11-02 09:33:38] ...trap/engines/engine.py.325:       elapsed: 0:10:39 | left: 0:29:21
[I 2022-11-02 09:33:38] ...trap/engines/engine.py.326:       process: 1.08033 | load: 0.00014
[I 2022-11-02 09:33:52] ...trap/engines/engine.py.321: eval: epoch 59 | batch 430/1522
[I 2022-11-02 09:33:52] ...trap/engines/engine.py.325:       elapsed: 0:10:54 | left: 0:25:24
[I 2022-11-02 09:33:52] ...trap/engines/engine.py.326:       process: 1.08404 | load: 0.00007
[I 2022-11-02 09:34:07] ...trap/engines/engine.py.321: eval: epoch 59 | batch 440/1522
[I 2022-11-02 09:34:07] ...trap/engines/engine.py.325:       elapsed: 0:11:09 | left: 0:27:28
[I 2022-11-02 09:34:07] ...trap/engines/engine.py.326:       process: 1.08231 | load: 0.00009
[I 2022-11-02 09:34:22] ...trap/engines/engine.py.321: eval: epoch 59 | batch 450/1522
[I 2022-11-02 09:34:22] ...trap/engines/engine.py.325:       elapsed: 0:11:24 | left: 0:26:23
[I 2022-11-02 09:34:22] ...trap/engines/engine.py.326:       process: 1.08263 | load: 0.00008
[I 2022-11-02 09:34:37] ...trap/engines/engine.py.321: eval: epoch 59 | batch 460/1522
[I 2022-11-02 09:34:37] ...trap/engines/engine.py.325:       elapsed: 0:11:39 | left: 0:25:59
[I 2022-11-02 09:34:37] ...trap/engines/engine.py.326:       process: 1.07531 | load: 0.00007
[I 2022-11-02 09:34:51] ...trap/engines/engine.py.321: eval: epoch 59 | batch 470/1522
[I 2022-11-02 09:34:51] ...trap/engines/engine.py.325:       elapsed: 0:11:52 | left: 0:22:38
[I 2022-11-02 09:34:51] ...trap/engines/engine.py.326:       process: 1.07860 | load: 0.00007
[I 2022-11-02 09:35:07] ...trap/engines/engine.py.321: eval: epoch 59 | batch 480/1522
[I 2022-11-02 09:35:07] ...trap/engines/engine.py.325:       elapsed: 0:12:09 | left: 0:28:18
[I 2022-11-02 09:35:07] ...trap/engines/engine.py.326:       process: 2.71058 | load: 1.61583
[I 2022-11-02 09:35:20] ...trap/engines/engine.py.321: eval: epoch 59 | batch 490/1522
[I 2022-11-02 09:35:20] ...trap/engines/engine.py.325:       elapsed: 0:12:22 | left: 0:23:28
[I 2022-11-02 09:35:20] ...trap/engines/engine.py.326:       process: 1.08112 | load: 0.00008
[I 2022-11-02 09:35:36] ...trap/engines/engine.py.321: eval: epoch 59 | batch 500/1522
[I 2022-11-02 09:35:36] ...trap/engines/engine.py.325:       elapsed: 0:12:37 | left: 0:24:17
[I 2022-11-02 09:35:36] ...trap/engines/engine.py.326:       process: 1.07921 | load: 0.00015
[I 2022-11-02 09:35:52] ...trap/engines/engine.py.321: eval: epoch 59 | batch 510/1522
[I 2022-11-02 09:35:52] ...trap/engines/engine.py.325:       elapsed: 0:12:54 | left: 0:26:47
[I 2022-11-02 09:35:52] ...trap/engines/engine.py.326:       process: 1.08658 | load: 0.00013
[I 2022-11-02 09:36:06] ...trap/engines/engine.py.321: eval: epoch 59 | batch 520/1522
[I 2022-11-02 09:36:06] ...trap/engines/engine.py.325:       elapsed: 0:13:08 | left: 0:24:45
[I 2022-11-02 09:36:06] ...trap/engines/engine.py.326:       process: 1.86290 | load: 0.77710
[I 2022-11-02 09:36:23] ...trap/engines/engine.py.321: eval: epoch 59 | batch 530/1522
[I 2022-11-02 09:36:23] ...trap/engines/engine.py.325:       elapsed: 0:13:25 | left: 0:29:08
[I 2022-11-02 09:36:23] ...trap/engines/engine.py.326:       process: 1.07666 | load: 0.00007
[I 2022-11-02 09:36:37] ...trap/engines/engine.py.321: eval: epoch 59 | batch 540/1522
[I 2022-11-02 09:36:37] ...trap/engines/engine.py.325:       elapsed: 0:13:38 | left: 0:24:39
[I 2022-11-02 09:36:37] ...trap/engines/engine.py.326:       process: 2.30389 | load: 1.21914
[I 2022-11-02 09:36:52] ...trap/engines/engine.py.321: eval: epoch 59 | batch 550/1522
[I 2022-11-02 09:36:52] ...trap/engines/engine.py.325:       elapsed: 0:13:53 | left: 0:24:29
[I 2022-11-02 09:36:52] ...trap/engines/engine.py.326:       process: 1.08507 | load: 0.00013
[I 2022-11-02 09:37:06] ...trap/engines/engine.py.321: eval: epoch 59 | batch 560/1522
[I 2022-11-02 09:37:06] ...trap/engines/engine.py.325:       elapsed: 0:14:08 | left: 0:24:03
[I 2022-11-02 09:37:06] ...trap/engines/engine.py.326:       process: 2.04258 | load: 0.96590
[I 2022-11-02 09:37:19] ...trap/engines/engine.py.321: eval: epoch 59 | batch 570/1522
[I 2022-11-02 09:37:19] ...trap/engines/engine.py.325:       elapsed: 0:14:21 | left: 0:21:48
[I 2022-11-02 09:37:19] ...trap/engines/engine.py.326:       process: 1.08145 | load: 0.00013
[I 2022-11-02 09:37:34] ...trap/engines/engine.py.321: eval: epoch 59 | batch 580/1522
[I 2022-11-02 09:37:34] ...trap/engines/engine.py.325:       elapsed: 0:14:35 | left: 0:21:57
[I 2022-11-02 09:37:34] ...trap/engines/engine.py.326:       process: 1.40565 | load: 0.32750
[I 2022-11-02 09:37:50] ...trap/engines/engine.py.321: eval: epoch 59 | batch 590/1522
[I 2022-11-02 09:37:50] ...trap/engines/engine.py.325:       elapsed: 0:14:52 | left: 0:25:23
[I 2022-11-02 09:37:50] ...trap/engines/engine.py.326:       process: 1.08412 | load: 0.00013
[I 2022-11-02 09:38:04] ...trap/engines/engine.py.321: eval: epoch 59 | batch 600/1522
[I 2022-11-02 09:38:04] ...trap/engines/engine.py.325:       elapsed: 0:15:06 | left: 0:20:58
[I 2022-11-02 09:38:04] ...trap/engines/engine.py.326:       process: 1.09031 | load: 0.00016
[I 2022-11-02 09:38:18] ...trap/engines/engine.py.321: eval: epoch 59 | batch 610/1522
[I 2022-11-02 09:38:18] ...trap/engines/engine.py.325:       elapsed: 0:15:20 | left: 0:20:03
[I 2022-11-02 09:38:18] ...trap/engines/engine.py.326:       process: 1.08527 | load: 0.00007
[I 2022-11-02 09:38:32] ...trap/engines/engine.py.321: eval: epoch 59 | batch 620/1522
[I 2022-11-02 09:38:32] ...trap/engines/engine.py.325:       elapsed: 0:15:33 | left: 0:19:38
[I 2022-11-02 09:38:32] ...trap/engines/engine.py.326:       process: 1.08026 | load: 0.00015
[I 2022-11-02 09:38:47] ...trap/engines/engine.py.321: eval: epoch 59 | batch 630/1522
[I 2022-11-02 09:38:47] ...trap/engines/engine.py.325:       elapsed: 0:15:49 | left: 0:21:49
[I 2022-11-02 09:38:47] ...trap/engines/engine.py.326:       process: 1.08502 | load: 0.00013
[I 2022-11-02 09:39:02] ...trap/engines/engine.py.321: eval: epoch 59 | batch 640/1522
[I 2022-11-02 09:39:02] ...trap/engines/engine.py.325:       elapsed: 0:16:03 | left: 0:20:15
[I 2022-11-02 09:39:02] ...trap/engines/engine.py.326:       process: 1.07887 | load: 0.00015
[I 2022-11-02 09:39:17] ...trap/engines/engine.py.321: eval: epoch 59 | batch 650/1522
[I 2022-11-02 09:39:17] ...trap/engines/engine.py.325:       elapsed: 0:16:19 | left: 0:22:14
[I 2022-11-02 09:39:17] ...trap/engines/engine.py.326:       process: 1.08302 | load: 0.00012
[I 2022-11-02 09:39:31] ...trap/engines/engine.py.321: eval: epoch 59 | batch 660/1522
[I 2022-11-02 09:39:31] ...trap/engines/engine.py.325:       elapsed: 0:16:33 | left: 0:19:05
[I 2022-11-02 09:39:31] ...trap/engines/engine.py.326:       process: 1.08768 | load: 0.00016
[I 2022-11-02 09:39:47] ...trap/engines/engine.py.321: eval: epoch 59 | batch 670/1522
[I 2022-11-02 09:39:47] ...trap/engines/engine.py.325:       elapsed: 0:16:48 | left: 0:22:26
[I 2022-11-02 09:39:47] ...trap/engines/engine.py.326:       process: 1.08082 | load: 0.00013
[I 2022-11-02 09:40:00] ...trap/engines/engine.py.321: eval: epoch 59 | batch 680/1522
[I 2022-11-02 09:40:00] ...trap/engines/engine.py.325:       elapsed: 0:17:02 | left: 0:18:21
[I 2022-11-02 09:40:00] ...trap/engines/engine.py.326:       process: 1.08505 | load: 0.00013
[I 2022-11-02 09:40:14] ...trap/engines/engine.py.321: eval: epoch 59 | batch 690/1522
[I 2022-11-02 09:40:14] ...trap/engines/engine.py.325:       elapsed: 0:17:16 | left: 0:19:47
[I 2022-11-02 09:40:14] ...trap/engines/engine.py.326:       process: 1.08688 | load: 0.00013
[I 2022-11-02 09:40:28] ...trap/engines/engine.py.321: eval: epoch 59 | batch 700/1522
[I 2022-11-02 09:40:28] ...trap/engines/engine.py.325:       elapsed: 0:17:30 | left: 0:17:56
[I 2022-11-02 09:40:28] ...trap/engines/engine.py.326:       process: 1.07821 | load: 0.00013
[I 2022-11-02 09:40:44] ...trap/engines/engine.py.321: eval: epoch 59 | batch 710/1522
[I 2022-11-02 09:40:44] ...trap/engines/engine.py.325:       elapsed: 0:17:46 | left: 0:18:45
[I 2022-11-02 09:40:44] ...trap/engines/engine.py.326:       process: 1.07501 | load: 0.00013
[I 2022-11-02 09:40:58] ...trap/engines/engine.py.321: eval: epoch 59 | batch 720/1522
[I 2022-11-02 09:40:58] ...trap/engines/engine.py.325:       elapsed: 0:18:00 | left: 0:19:01
[I 2022-11-02 09:40:58] ...trap/engines/engine.py.326:       process: 1.09130 | load: 0.00014
[I 2022-11-02 09:41:14] ...trap/engines/engine.py.321: eval: epoch 59 | batch 730/1522
[I 2022-11-02 09:41:14] ...trap/engines/engine.py.325:       elapsed: 0:18:16 | left: 0:19:22
[I 2022-11-02 09:41:14] ...trap/engines/engine.py.326:       process: 1.07810 | load: 0.00007
[I 2022-11-02 09:41:28] ...trap/engines/engine.py.321: eval: epoch 59 | batch 740/1522
[I 2022-11-02 09:41:28] ...trap/engines/engine.py.325:       elapsed: 0:18:30 | left: 0:17:50
[I 2022-11-02 09:41:28] ...trap/engines/engine.py.326:       process: 1.08274 | load: 0.00014
[I 2022-11-02 09:41:44] ...trap/engines/engine.py.321: eval: epoch 59 | batch 750/1522
[I 2022-11-02 09:41:44] ...trap/engines/engine.py.325:       elapsed: 0:18:46 | left: 0:21:07
[I 2022-11-02 09:41:44] ...trap/engines/engine.py.326:       process: 1.08811 | load: 0.00007
[I 2022-11-02 09:41:58] ...trap/engines/engine.py.321: eval: epoch 59 | batch 760/1522
[I 2022-11-02 09:41:58] ...trap/engines/engine.py.325:       elapsed: 0:18:59 | left: 0:17:14
[I 2022-11-02 09:41:58] ...trap/engines/engine.py.326:       process: 1.08911 | load: 0.00013
[I 2022-11-02 09:42:14] ...trap/engines/engine.py.321: eval: epoch 59 | batch 770/1522
[I 2022-11-02 09:42:14] ...trap/engines/engine.py.325:       elapsed: 0:19:16 | left: 0:19:50
[I 2022-11-02 09:42:14] ...trap/engines/engine.py.326:       process: 1.09387 | load: 0.00007
[I 2022-11-02 09:42:29] ...trap/engines/engine.py.321: eval: epoch 59 | batch 780/1522
[I 2022-11-02 09:42:29] ...trap/engines/engine.py.325:       elapsed: 0:19:30 | left: 0:16:59
[I 2022-11-02 09:42:29] ...trap/engines/engine.py.326:       process: 1.08318 | load: 0.00014
[I 2022-11-02 09:42:44] ...trap/engines/engine.py.321: eval: epoch 59 | batch 790/1522
[I 2022-11-02 09:42:44] ...trap/engines/engine.py.325:       elapsed: 0:19:46 | left: 0:17:44
[I 2022-11-02 09:42:44] ...trap/engines/engine.py.326:       process: 1.08432 | load: 0.00007
[I 2022-11-02 09:42:58] ...trap/engines/engine.py.321: eval: epoch 59 | batch 800/1522
[I 2022-11-02 09:42:58] ...trap/engines/engine.py.325:       elapsed: 0:20:00 | left: 0:15:47
[I 2022-11-02 09:42:58] ...trap/engines/engine.py.326:       process: 1.08242 | load: 0.00013
[I 2022-11-02 09:43:13] ...trap/engines/engine.py.321: eval: epoch 59 | batch 810/1522
[I 2022-11-02 09:43:13] ...trap/engines/engine.py.325:       elapsed: 0:20:15 | left: 0:17:53
[I 2022-11-02 09:43:13] ...trap/engines/engine.py.326:       process: 1.08778 | load: 0.00007
[I 2022-11-02 09:43:26] ...trap/engines/engine.py.321: eval: epoch 59 | batch 820/1522
[I 2022-11-02 09:43:26] ...trap/engines/engine.py.325:       elapsed: 0:20:28 | left: 0:15:20
[I 2022-11-02 09:43:26] ...trap/engines/engine.py.326:       process: 1.08626 | load: 0.00013
[I 2022-11-02 09:43:41] ...trap/engines/engine.py.321: eval: epoch 59 | batch 830/1522
[I 2022-11-02 09:43:41] ...trap/engines/engine.py.325:       elapsed: 0:20:43 | left: 0:17:28
[I 2022-11-02 09:43:41] ...trap/engines/engine.py.326:       process: 1.08626 | load: 0.00007
[I 2022-11-02 09:43:54] ...trap/engines/engine.py.321: eval: epoch 59 | batch 840/1522
[I 2022-11-02 09:43:54] ...trap/engines/engine.py.325:       elapsed: 0:20:56 | left: 0:14:38
[I 2022-11-02 09:43:54] ...trap/engines/engine.py.326:       process: 1.08997 | load: 0.00014
[I 2022-11-02 09:44:10] ...trap/engines/engine.py.321: eval: epoch 59 | batch 850/1522
[I 2022-11-02 09:44:10] ...trap/engines/engine.py.325:       elapsed: 0:21:12 | left: 0:16:33
[I 2022-11-02 09:44:10] ...trap/engines/engine.py.326:       process: 1.08809 | load: 0.00007
[I 2022-11-02 09:44:25] ...trap/engines/engine.py.321: eval: epoch 59 | batch 860/1522
[I 2022-11-02 09:44:25] ...trap/engines/engine.py.325:       elapsed: 0:21:26 | left: 0:15:05
[I 2022-11-02 09:44:25] ...trap/engines/engine.py.326:       process: 1.08648 | load: 0.00013
[I 2022-11-02 09:44:41] ...trap/engines/engine.py.321: eval: epoch 59 | batch 870/1522
[I 2022-11-02 09:44:41] ...trap/engines/engine.py.325:       elapsed: 0:21:43 | left: 0:17:25
[I 2022-11-02 09:44:41] ...trap/engines/engine.py.326:       process: 1.08366 | load: 0.00007
[I 2022-11-02 09:44:55] ...trap/engines/engine.py.321: eval: epoch 59 | batch 880/1522
[I 2022-11-02 09:44:55] ...trap/engines/engine.py.325:       elapsed: 0:21:56 | left: 0:14:32
[I 2022-11-02 09:44:55] ...trap/engines/engine.py.326:       process: 1.09302 | load: 0.00013
[I 2022-11-02 09:45:11] ...trap/engines/engine.py.321: eval: epoch 59 | batch 890/1522
[I 2022-11-02 09:45:11] ...trap/engines/engine.py.325:       elapsed: 0:22:13 | left: 0:16:35
[I 2022-11-02 09:45:11] ...trap/engines/engine.py.326:       process: 1.08982 | load: 0.00007
[I 2022-11-02 09:45:25] ...trap/engines/engine.py.321: eval: epoch 59 | batch 900/1522
[I 2022-11-02 09:45:25] ...trap/engines/engine.py.325:       elapsed: 0:22:27 | left: 0:14:10
[I 2022-11-02 09:45:25] ...trap/engines/engine.py.326:       process: 1.08701 | load: 0.00014
[I 2022-11-02 09:45:40] ...trap/engines/engine.py.321: eval: epoch 59 | batch 910/1522
[I 2022-11-02 09:45:40] ...trap/engines/engine.py.325:       elapsed: 0:22:42 | left: 0:14:40
[I 2022-11-02 09:45:40] ...trap/engines/engine.py.326:       process: 1.07642 | load: 0.00007
[I 2022-11-02 09:45:54] ...trap/engines/engine.py.321: eval: epoch 59 | batch 920/1522
[I 2022-11-02 09:45:54] ...trap/engines/engine.py.325:       elapsed: 0:22:55 | left: 0:13:15
[I 2022-11-02 09:45:54] ...trap/engines/engine.py.326:       process: 1.08241 | load: 0.00013
[I 2022-11-02 09:46:09] ...trap/engines/engine.py.321: eval: epoch 59 | batch 930/1522
[I 2022-11-02 09:46:09] ...trap/engines/engine.py.325:       elapsed: 0:23:11 | left: 0:15:10
[I 2022-11-02 09:46:09] ...trap/engines/engine.py.326:       process: 1.07929 | load: 0.00007
[I 2022-11-02 09:46:24] ...trap/engines/engine.py.321: eval: epoch 59 | batch 940/1522
[I 2022-11-02 09:46:24] ...trap/engines/engine.py.325:       elapsed: 0:23:25 | left: 0:13:20
[I 2022-11-02 09:46:24] ...trap/engines/engine.py.326:       process: 1.08708 | load: 0.00014
[I 2022-11-02 09:46:39] ...trap/engines/engine.py.321: eval: epoch 59 | batch 950/1522
[I 2022-11-02 09:46:39] ...trap/engines/engine.py.325:       elapsed: 0:23:41 | left: 0:14:24
[I 2022-11-02 09:46:39] ...trap/engines/engine.py.326:       process: 1.09553 | load: 0.00013
[I 2022-11-02 09:46:54] ...trap/engines/engine.py.321: eval: epoch 59 | batch 960/1522
[I 2022-11-02 09:46:54] ...trap/engines/engine.py.325:       elapsed: 0:23:56 | left: 0:13:25
[I 2022-11-02 09:46:54] ...trap/engines/engine.py.326:       process: 1.09337 | load: 0.00008
[I 2022-11-02 09:47:11] ...trap/engines/engine.py.321: eval: epoch 59 | batch 970/1522
[I 2022-11-02 09:47:11] ...trap/engines/engine.py.325:       elapsed: 0:24:13 | left: 0:15:30
[I 2022-11-02 09:47:11] ...trap/engines/engine.py.326:       process: 1.08052 | load: 0.00013
[I 2022-11-02 09:47:25] ...trap/engines/engine.py.321: eval: epoch 59 | batch 980/1522
[I 2022-11-02 09:47:25] ...trap/engines/engine.py.325:       elapsed: 0:24:27 | left: 0:12:52
[I 2022-11-02 09:47:25] ...trap/engines/engine.py.326:       process: 1.08042 | load: 0.00013
[I 2022-11-02 09:47:41] ...trap/engines/engine.py.321: eval: epoch 59 | batch 990/1522
[I 2022-11-02 09:47:41] ...trap/engines/engine.py.325:       elapsed: 0:24:43 | left: 0:14:16
[I 2022-11-02 09:47:41] ...trap/engines/engine.py.326:       process: 1.08276 | load: 0.00007
[I 2022-11-02 09:48:07] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1000/1522
[I 2022-11-02 09:48:07] ...trap/engines/engine.py.325:       elapsed: 0:25:09 | left: 0:26:22
[I 2022-11-02 09:48:07] ...trap/engines/engine.py.326:       process: 1.07897 | load: 0.00007
[I 2022-11-02 09:48:35] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1010/1522
[I 2022-11-02 09:48:35] ...trap/engines/engine.py.325:       elapsed: 0:25:37 | left: 0:19:02
[I 2022-11-02 09:48:35] ...trap/engines/engine.py.326:       process: 1.07484 | load: 0.00007
[I 2022-11-02 09:48:52] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1020/1522
[I 2022-11-02 09:48:52] ...trap/engines/engine.py.325:       elapsed: 0:25:54 | left: 0:13:58
[I 2022-11-02 09:48:52] ...trap/engines/engine.py.326:       process: 1.07860 | load: 0.00007
[I 2022-11-02 09:49:13] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1030/1522
[I 2022-11-02 09:49:13] ...trap/engines/engine.py.325:       elapsed: 0:26:15 | left: 0:18:38
[I 2022-11-02 09:49:13] ...trap/engines/engine.py.326:       process: 1.08182 | load: 0.00013
[I 2022-11-02 09:49:33] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1040/1522
[I 2022-11-02 09:49:33] ...trap/engines/engine.py.325:       elapsed: 0:26:34 | left: 0:14:31
[I 2022-11-02 09:49:33] ...trap/engines/engine.py.326:       process: 1.07490 | load: 0.00007
[I 2022-11-02 09:49:51] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1050/1522
[I 2022-11-02 09:49:51] ...trap/engines/engine.py.325:       elapsed: 0:26:53 | left: 0:13:40
[I 2022-11-02 09:49:51] ...trap/engines/engine.py.326:       process: 1.08106 | load: 0.00007
[I 2022-11-02 09:50:07] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1060/1522
[I 2022-11-02 09:50:07] ...trap/engines/engine.py.325:       elapsed: 0:27:09 | left: 0:12:03
[I 2022-11-02 09:50:07] ...trap/engines/engine.py.326:       process: 1.08593 | load: 0.00007
[I 2022-11-02 09:50:24] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1070/1522
[I 2022-11-02 09:50:24] ...trap/engines/engine.py.325:       elapsed: 0:27:26 | left: 0:11:36
[I 2022-11-02 09:50:24] ...trap/engines/engine.py.326:       process: 1.08612 | load: 0.00007
[I 2022-11-02 09:50:41] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1080/1522
[I 2022-11-02 09:50:41] ...trap/engines/engine.py.325:       elapsed: 0:27:43 | left: 0:12:32
[I 2022-11-02 09:50:41] ...trap/engines/engine.py.326:       process: 1.07684 | load: 0.00012
[I 2022-11-02 09:50:57] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1090/1522
[I 2022-11-02 09:50:57] ...trap/engines/engine.py.325:       elapsed: 0:27:59 | left: 0:10:50
[I 2022-11-02 09:50:57] ...trap/engines/engine.py.326:       process: 1.08296 | load: 0.00014
[I 2022-11-02 09:51:13] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1100/1522
[I 2022-11-02 09:51:13] ...trap/engines/engine.py.325:       elapsed: 0:28:15 | left: 0:10:54
[I 2022-11-02 09:51:13] ...trap/engines/engine.py.326:       process: 1.09193 | load: 0.00007
[I 2022-11-02 09:51:27] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1110/1522
[I 2022-11-02 09:51:27] ...trap/engines/engine.py.325:       elapsed: 0:28:29 | left: 0:09:12
[I 2022-11-02 09:51:27] ...trap/engines/engine.py.326:       process: 1.08386 | load: 0.00007
[I 2022-11-02 09:51:42] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1120/1522
[I 2022-11-02 09:51:42] ...trap/engines/engine.py.325:       elapsed: 0:28:44 | left: 0:09:53
[I 2022-11-02 09:51:42] ...trap/engines/engine.py.326:       process: 1.08505 | load: 0.00007
[I 2022-11-02 09:51:58] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1130/1522
[I 2022-11-02 09:51:58] ...trap/engines/engine.py.325:       elapsed: 0:29:00 | left: 0:10:17
[I 2022-11-02 09:51:58] ...trap/engines/engine.py.326:       process: 1.10347 | load: 0.00007
[I 2022-11-02 09:52:14] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1140/1522
[I 2022-11-02 09:52:14] ...trap/engines/engine.py.325:       elapsed: 0:29:15 | left: 0:09:47
[I 2022-11-02 09:52:14] ...trap/engines/engine.py.326:       process: 1.08310 | load: 0.00007
[I 2022-11-02 09:52:29] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1150/1522
[I 2022-11-02 09:52:29] ...trap/engines/engine.py.325:       elapsed: 0:29:31 | left: 0:08:40
[I 2022-11-02 09:52:29] ...trap/engines/engine.py.326:       process: 1.06689 | load: 0.00007
[I 2022-11-02 09:52:46] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1160/1522
[I 2022-11-02 09:52:46] ...trap/engines/engine.py.325:       elapsed: 0:29:48 | left: 0:10:23
[I 2022-11-02 09:52:46] ...trap/engines/engine.py.326:       process: 1.07630 | load: 0.00007
[I 2022-11-02 09:53:00] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1170/1522
[I 2022-11-02 09:53:00] ...trap/engines/engine.py.325:       elapsed: 0:30:02 | left: 0:07:48
[I 2022-11-02 09:53:00] ...trap/engines/engine.py.326:       process: 1.07913 | load: 0.00013
[I 2022-11-02 09:53:16] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1180/1522
[I 2022-11-02 09:53:16] ...trap/engines/engine.py.325:       elapsed: 0:30:18 | left: 0:09:03
[I 2022-11-02 09:53:16] ...trap/engines/engine.py.326:       process: 1.08086 | load: 0.00013
[I 2022-11-02 09:53:30] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1190/1522
[I 2022-11-02 09:53:30] ...trap/engines/engine.py.325:       elapsed: 0:30:32 | left: 0:07:34
[I 2022-11-02 09:53:30] ...trap/engines/engine.py.326:       process: 1.09146 | load: 0.00016
[I 2022-11-02 09:53:46] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1200/1522
[I 2022-11-02 09:53:46] ...trap/engines/engine.py.325:       elapsed: 0:30:47 | left: 0:07:55
[I 2022-11-02 09:53:46] ...trap/engines/engine.py.326:       process: 1.08324 | load: 0.00013
[I 2022-11-02 09:53:59] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1210/1522
[I 2022-11-02 09:53:59] ...trap/engines/engine.py.325:       elapsed: 0:31:01 | left: 0:06:46
[I 2022-11-02 09:53:59] ...trap/engines/engine.py.326:       process: 1.10492 | load: 0.00015
[I 2022-11-02 09:54:15] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1220/1522
[I 2022-11-02 09:54:15] ...trap/engines/engine.py.325:       elapsed: 0:31:17 | left: 0:08:11
[I 2022-11-02 09:54:15] ...trap/engines/engine.py.326:       process: 1.11043 | load: 0.00013
[I 2022-11-02 09:54:30] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1230/1522
[I 2022-11-02 09:54:30] ...trap/engines/engine.py.325:       elapsed: 0:31:32 | left: 0:06:49
[I 2022-11-02 09:54:30] ...trap/engines/engine.py.326:       process: 1.08317 | load: 0.00016
[I 2022-11-02 09:54:45] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1240/1522
[I 2022-11-02 09:54:45] ...trap/engines/engine.py.325:       elapsed: 0:31:46 | left: 0:06:43
[I 2022-11-02 09:54:45] ...trap/engines/engine.py.326:       process: 1.07808 | load: 0.00007
[I 2022-11-02 09:54:59] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1250/1522
[I 2022-11-02 09:54:59] ...trap/engines/engine.py.325:       elapsed: 0:32:00 | left: 0:06:17
[I 2022-11-02 09:54:59] ...trap/engines/engine.py.326:       process: 1.08185 | load: 0.00016
[I 2022-11-02 09:55:13] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1260/1522
[I 2022-11-02 09:55:13] ...trap/engines/engine.py.325:       elapsed: 0:32:14 | left: 0:06:03
[I 2022-11-02 09:55:13] ...trap/engines/engine.py.326:       process: 1.08490 | load: 0.00013
[I 2022-11-02 09:55:26] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1270/1522
[I 2022-11-02 09:55:26] ...trap/engines/engine.py.325:       elapsed: 0:32:28 | left: 0:05:27
[I 2022-11-02 09:55:26] ...trap/engines/engine.py.326:       process: 1.08179 | load: 0.00015
[I 2022-11-02 09:55:41] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1280/1522
[I 2022-11-02 09:55:41] ...trap/engines/engine.py.325:       elapsed: 0:32:43 | left: 0:05:45
[I 2022-11-02 09:55:41] ...trap/engines/engine.py.326:       process: 1.08619 | load: 0.00016
[I 2022-11-02 09:55:55] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1290/1522
[I 2022-11-02 09:55:55] ...trap/engines/engine.py.325:       elapsed: 0:32:57 | left: 0:05:53
[I 2022-11-02 09:55:55] ...trap/engines/engine.py.326:       process: 1.27568 | load: 0.19818
[I 2022-11-02 09:56:10] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1300/1522
[I 2022-11-02 09:56:10] ...trap/engines/engine.py.325:       elapsed: 0:33:11 | left: 0:05:08
[I 2022-11-02 09:56:10] ...trap/engines/engine.py.326:       process: 1.12607 | load: 0.00016
[I 2022-11-02 09:56:26] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1310/1522
[I 2022-11-02 09:56:26] ...trap/engines/engine.py.325:       elapsed: 0:33:28 | left: 0:05:34
[I 2022-11-02 09:56:26] ...trap/engines/engine.py.326:       process: 1.06912 | load: 0.00016
[I 2022-11-02 09:56:40] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1320/1522
[I 2022-11-02 09:56:40] ...trap/engines/engine.py.325:       elapsed: 0:33:42 | left: 0:04:43
[I 2022-11-02 09:56:40] ...trap/engines/engine.py.326:       process: 1.08244 | load: 0.00007
[I 2022-11-02 09:56:57] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1330/1522
[I 2022-11-02 09:56:57] ...trap/engines/engine.py.325:       elapsed: 0:33:58 | left: 0:04:56
[I 2022-11-02 09:56:57] ...trap/engines/engine.py.326:       process: 1.09162 | load: 0.00013
[I 2022-11-02 09:57:10] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1340/1522
[I 2022-11-02 09:57:10] ...trap/engines/engine.py.325:       elapsed: 0:34:12 | left: 0:04:06
[I 2022-11-02 09:57:10] ...trap/engines/engine.py.326:       process: 1.09195 | load: 0.00007
[I 2022-11-02 09:57:25] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1350/1522
[I 2022-11-02 09:57:25] ...trap/engines/engine.py.325:       elapsed: 0:34:27 | left: 0:04:05
[I 2022-11-02 09:57:25] ...trap/engines/engine.py.326:       process: 1.08357 | load: 0.00013
[I 2022-11-02 09:57:40] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1360/1522
[I 2022-11-02 09:57:40] ...trap/engines/engine.py.325:       elapsed: 0:34:42 | left: 0:03:40
[I 2022-11-02 09:57:40] ...trap/engines/engine.py.326:       process: 1.07132 | load: 0.00007
[I 2022-11-02 09:57:55] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1370/1522
[I 2022-11-02 09:57:55] ...trap/engines/engine.py.325:       elapsed: 0:34:57 | left: 0:03:44
[I 2022-11-02 09:57:55] ...trap/engines/engine.py.326:       process: 1.08300 | load: 0.00007
[I 2022-11-02 09:58:08] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1380/1522
[I 2022-11-02 09:58:08] ...trap/engines/engine.py.325:       elapsed: 0:35:10 | left: 0:03:00
[I 2022-11-02 09:58:08] ...trap/engines/engine.py.326:       process: 1.08329 | load: 0.00011
[I 2022-11-02 09:58:24] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1390/1522
[I 2022-11-02 09:58:24] ...trap/engines/engine.py.325:       elapsed: 0:35:25 | left: 0:03:14
[I 2022-11-02 09:58:24] ...trap/engines/engine.py.326:       process: 1.09489 | load: 0.00014
[I 2022-11-02 09:58:37] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1400/1522
[I 2022-11-02 09:58:37] ...trap/engines/engine.py.325:       elapsed: 0:35:39 | left: 0:02:40
[I 2022-11-02 09:58:37] ...trap/engines/engine.py.326:       process: 1.08752 | load: 0.00007
[I 2022-11-02 09:58:53] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1410/1522
[I 2022-11-02 09:58:53] ...trap/engines/engine.py.325:       elapsed: 0:35:54 | left: 0:03:03
[I 2022-11-02 09:58:53] ...trap/engines/engine.py.326:       process: 2.38162 | load: 1.29945
[I 2022-11-02 09:59:06] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1420/1522
[I 2022-11-02 09:59:06] ...trap/engines/engine.py.325:       elapsed: 0:36:08 | left: 0:02:26
[I 2022-11-02 09:59:06] ...trap/engines/engine.py.326:       process: 1.08424 | load: 0.00007
[I 2022-11-02 09:59:22] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1430/1522
[I 2022-11-02 09:59:22] ...trap/engines/engine.py.325:       elapsed: 0:36:23 | left: 0:02:30
[I 2022-11-02 09:59:22] ...trap/engines/engine.py.326:       process: 1.07757 | load: 0.00015
[I 2022-11-02 09:59:35] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1440/1522
[I 2022-11-02 09:59:35] ...trap/engines/engine.py.325:       elapsed: 0:36:37 | left: 0:01:48
[I 2022-11-02 09:59:35] ...trap/engines/engine.py.326:       process: 1.09432 | load: 0.00015
[I 2022-11-02 09:59:50] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1450/1522
[I 2022-11-02 09:59:50] ...trap/engines/engine.py.325:       elapsed: 0:36:52 | left: 0:01:43
[I 2022-11-02 09:59:50] ...trap/engines/engine.py.326:       process: 1.09626 | load: 0.00013
[I 2022-11-02 10:00:04] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1460/1522
[I 2022-11-02 10:00:04] ...trap/engines/engine.py.325:       elapsed: 0:37:06 | left: 0:01:31
[I 2022-11-02 10:00:04] ...trap/engines/engine.py.326:       process: 2.16281 | load: 1.07773
[I 2022-11-02 10:00:18] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1470/1522
[I 2022-11-02 10:00:18] ...trap/engines/engine.py.325:       elapsed: 0:37:20 | left: 0:01:14
[I 2022-11-02 10:00:18] ...trap/engines/engine.py.326:       process: 1.08381 | load: 0.00007
[I 2022-11-02 10:00:33] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1480/1522
[I 2022-11-02 10:00:33] ...trap/engines/engine.py.325:       elapsed: 0:37:35 | left: 0:01:04
[I 2022-11-02 10:00:33] ...trap/engines/engine.py.326:       process: 1.08158 | load: 0.00016
[I 2022-11-02 10:00:46] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1490/1522
[I 2022-11-02 10:00:46] ...trap/engines/engine.py.325:       elapsed: 0:37:48 | left: 0:00:40
[I 2022-11-02 10:00:46] ...trap/engines/engine.py.326:       process: 1.08901 | load: 0.00007
[I 2022-11-02 10:01:01] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1500/1522
[I 2022-11-02 10:01:01] ...trap/engines/engine.py.325:       elapsed: 0:38:03 | left: 0:00:30
[I 2022-11-02 10:01:01] ...trap/engines/engine.py.326:       process: 1.32329 | load: 0.23424
[I 2022-11-02 10:01:16] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1510/1522
[I 2022-11-02 10:01:16] ...trap/engines/engine.py.325:       elapsed: 0:38:17 | left: 0:00:16
[I 2022-11-02 10:01:16] ...trap/engines/engine.py.326:       process: 1.07957 | load: 0.00007
[I 2022-11-02 10:01:32] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1520/1522
[I 2022-11-02 10:01:32] ...trap/engines/engine.py.325:       elapsed: 0:38:34 | left: 0:00:02
[I 2022-11-02 10:01:32] ...trap/engines/engine.py.326:       process: 1.10271 | load: 0.00002
[S 2022-11-02 10:01:35] ...trap/engines/engine.py.113: eval_epoch.epoch: 59
[I 2022-11-02 10:01:35] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs
[S 2022-11-02 10:01:58] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 2.00
[S 2022-11-02 10:01:58] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 2.00
[S 2022-11-02 10:01:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.473000
[S 2022-11-02 10:01:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.775500
[S 2022-11-02 10:01:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.853000
[S 2022-11-02 10:01:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.452300
[S 2022-11-02 10:01:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.758800
[S 2022-11-02 10:01:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.840100
[S 2022-11-02 10:01:58] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000
[S 2022-11-02 10:01:58] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000
[S 2022-11-02 10:01:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.007836
[S 2022-11-02 10:01:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.014334
[S 2022-11-02 10:01:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.012530
[S 2022-11-02 10:01:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.010070
[S 2022-11-02 10:01:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.011338
[S 2022-11-02 10:01:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.013003
[I 2022-11-02 10:02:10] ...trap/engines/engine.py.115: Ending evaluation procedures
[W 2022-11-02 10:02:15] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json'
[W 2022-11-02 10:02:15] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[W 2022-11-02 10:02:15] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[I 2022-11-02 10:02:26] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_recipe1m_13m_orig_ids/view.html
[I 2022-11-10 17:59:37] ...ytorch/bootstrap/run.py.73: Saving environment info
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75: options: 
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:   dataset:
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     aux_kw_path:
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       test: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_recipe1m_13m_orig_test_titles_kw.json
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       train: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_recipe1m_13m_train_titles_kw.json
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       val: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_recipe1m_13m_orig_val_titles_kw.json
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     batch_sampler: random
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     batch_size: 100
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     database: lmdb
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     dir: /gpfsdswork/dataset/Recipe1M+/images_recipe1M+
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     eval_split: test
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     freq_mismatch: 0.0
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     image_from: database
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     import: recipe1m.datasets.factory
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     kw_path:
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       test: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_recipe1m_13m_orig_test_ingr_kw.json
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       train: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_recipe1m_13m_train_ingr_kw.json
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       val: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_recipe1m_13m_orig_val_ingr_kw.json
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     name: recipe1m_13m
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     nb_classes: 1048
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     nb_threads: 4
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     orig_ids_eval: True
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     path_ids: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/recipe1m_13m/original_ids.json
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     path_image_json: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/recipe1m_13m/layer2+.json
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     path_text_json: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/text/tokenized_layer1.json
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     randkw_p: 0.3
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     randkw_p_aux: 0.5
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     tokenized_raw_text: True
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     train_split: None
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     use_vcs: True
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     vocab_path: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/text/vocab_all.txt
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:   engine:
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     name: logger
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     nb_epochs: 60
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     print_freq: 10
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     saving_criteria: ['eval_epoch.metric.med_im2recipe_mean:min', 'eval_epoch.metric.recall_at_1_im2recipe_mean:max']
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:   exp:
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     dir: /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_recipe1m_13m_orig_ids
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     resume: best_eval_epoch.metric.recall_at_1_im2recipe_mean
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:   misc:
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     cuda: True
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     data_parrallel: True
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     logs_name: None
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     overrite: False
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     seed: 1338
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:   model:
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     criterion:
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:         tri_cross_encoder: False
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       itm_loss_weight: 1
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       keep_background: False
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       retrieval_strategy:
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:         aggregation: valid
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:         get_tokens: True
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:         margin: 0.05
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:         margin_params:
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:           increment: 0.005
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:           increment_margin: True
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:           max_margin: 0.3
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:         name: vlp_triplet
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:         nb_samples: 9999
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:         sampling: max_negative
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:         substrategy: ['IRR', 'RII']
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:         substrategy_weights: [1.0, 1.0]
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     import: recipe1m.models.factory
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     metric:
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       k_test: 10
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       nb_bags: 5
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       nb_matchs_per_bag: 10000
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       retrieval_dir: /gpfsscratch/rech/dyf/ugz83ue/tmp/recipe1m
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       trijoint: True
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     name: trijoint
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     network:
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       activations: ['tanh', 'normalize']
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       aux_kwords: True
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       aux_kwords_encoder: True
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       avg_concat: True
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       bert_config: /linkhome/rech/genisi01/ugz83ue/tfood/recipe1m/models/networks/recipe_networks/config_bert_albef.json
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       cat_pos: 0
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       class_attention: False
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       class_attention_fus: False
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       cls_token: False
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       cls_token_fus: False
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       cross_decoder_image: True
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       cross_decoder_recipe: True
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:         class_attention: False
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:         cls_token: False
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:         cross_attention: False
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:         cross_decoder: True
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:         double_decoder: False
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:         get_tokens: False
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:         load_backbones: False
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:         n_heads: 4
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:         n_layers: 4
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:         query_img: False
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       cross_transformer: False
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       dim_emb: 1024
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       dim_image_out: 2048
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       dim_ingr_out: 300
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       dim_instr_in: 1024
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       dim_instr_out: 1024
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       freeze_im: False
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       freeze_rec: False
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       get_tokens: True
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       get_tokens_cross_decoder_recipe: True
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       hidden_size: 512
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       image_backbone_name: clip
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       image_res: 224
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       keep_embedding: True
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       kwords_same_level: False
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       last_cat_aux: True
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       n_heads: 4
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       n_heads_cross: 4
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       n_heads_cross_image: 4
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       n_heads_single: 4
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       n_layers: 2
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       n_layers_cross: 2
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       n_layers_cross_image: 1
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       n_layers_single: 2
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       num_hidden_layers_kw: 2
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       old: True
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       path_ingrs: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/text/vocab.pkl
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       path_vocab: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/text/vocab_all.txt
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       recipe_encoder: tfoodvicha-transformer
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       text_encoder: bert-base-uncased
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       vision_width: 768
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       vit_all_tokens: True
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       with_ingrs: True
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       with_instrs: True
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:       with_titles: True
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     with_classif: False
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:   optimizer:
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     batch_size_factor: None
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     clip_grad: 8.0
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     import: recipe1m.optimizers.factory
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     lr: 1e-05
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     lr_img: 1e-06
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     name: trijoint_fixed_fine_tune
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:     switch_epoch: 5
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:   path_opts: /linkhome/rech/genisi01/ugz83ue/tfood/recipe1m/options/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_recipe1m_13m.yaml
[S 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.75:   view: ['logs:train_epoch.loss', 'logs:train_epoch.bad_pairs', 'logs:eval_epoch.metric.med_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_1_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_5_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_10_im2recipe_mean', 'logs:eval_epoch.metric.med_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_1_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_5_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_10_recipe2im_mean', 'logs:optimizer.is_optimizer_recipe&image', 'logs:optimizer.total_norm']
[I 2022-11-10 17:59:43] ...ytorch/bootstrap/run.py.76: posix.uname_result(sysname='Linux', nodename='jean-zay-ia801', release='4.18.0-305.65.1.el8_4.x86_64', version='#1 SMP Thu Sep 22 08:28:21 EDT 2022', machine='x86_64')
[I 2022-11-10 17:59:44] ...ytorch/bootstrap/run.py.80: Available GPUs: [0]
[I 2022-11-10 17:59:44] ...strap/engines/factory.py.9: Creating engine...
[I 2022-11-10 17:59:44] ...trap/datasets/factory.py.8: Creating dataset...
[I 2022-11-10 18:01:32] ...tasets/recipe1m_13m.py.290: Load VCs...
[I 2022-11-10 18:01:33] ...tasets/recipe1m_13m.py.303: randkw_p... None
[I 2022-11-10 18:01:33] ...rap/datasets/factory.py.21: Evaluation will take place on testset (152388 items)
[I 2022-11-10 18:01:33] ...strap/models/factory.py.21: Creating model...
[I 2022-11-10 18:01:47] ...e_networks/networks.py.203: Loading CLIP Visual model...
[I 2022-11-10 18:01:51] ...e_networks/networks.py.145: cuda
[I 2022-11-10 18:01:52] ...networks/tfood_vicha.py.48: Load VCs encoder...
[I 2022-11-10 18:01:53] ...els/metrics/trijoint.py.46: self.save_ids False
[I 2022-11-10 18:01:53] ...strap/models/factory.py.44: Enabling CUDA mode...
[I 2022-11-10 18:02:00] ...etworks/tfood_vicha.py.183: training aux kw modules
[I 2022-11-10 18:02:00] ...etworks/tfood_vicha.py.183: training aux kw modules
[I 2022-11-10 18:02:00] ...otstrap/views/factory.py.8: Creating views...
[I 2022-11-10 18:02:04] ...strap/engines/engine.py.98: Loading best_eval_epoch.metric.recall_at_1_im2recipe_mean checkpoint
[I 2022-11-10 18:02:04] ...trap/engines/engine.py.440: Loading model...
[I 2022-11-10 18:02:10] ...trap/engines/engine.py.453: Loading engine...
[I 2022-11-10 18:02:10] ...trap/engines/engine.py.108: Launching evaluation procedures
[I 2022-11-10 18:02:10] ...trap/engines/engine.py.268: Evaluating model on testset for epoch 59
[I 2022-11-10 18:02:10] ...tasets/recipe1m_13m.py.468: Dataset will be sampled with "random" batch_sampler.
[I 2022-11-10 18:02:20] ...trap/engines/engine.py.321: eval: epoch 59 | batch 0/1522
[I 2022-11-10 18:02:20] ...trap/engines/engine.py.325:       elapsed: 0:00:09 | left: 4:11:17
[I 2022-11-10 18:02:20] ...trap/engines/engine.py.326:       process: 9.90651 | load: 7.57140
[I 2022-11-10 18:02:35] ...trap/engines/engine.py.321: eval: epoch 59 | batch 10/1522
[I 2022-11-10 18:02:35] ...trap/engines/engine.py.325:       elapsed: 0:00:24 | left: 1:00:31
[I 2022-11-10 18:02:35] ...trap/engines/engine.py.326:       process: 1.18021 | load: 0.00012
[I 2022-11-10 18:02:53] ...trap/engines/engine.py.321: eval: epoch 59 | batch 20/1522
[I 2022-11-10 18:02:53] ...trap/engines/engine.py.325:       elapsed: 0:00:42 | left: 0:48:54
[I 2022-11-10 18:02:53] ...trap/engines/engine.py.326:       process: 3.19550 | load: 2.03044
[I 2022-11-10 18:03:08] ...trap/engines/engine.py.321: eval: epoch 59 | batch 30/1522
[I 2022-11-10 18:03:08] ...trap/engines/engine.py.325:       elapsed: 0:00:57 | left: 0:39:43
[I 2022-11-10 18:03:08] ...trap/engines/engine.py.326:       process: 1.15677 | load: 0.00014
[I 2022-11-10 18:03:25] ...trap/engines/engine.py.321: eval: epoch 59 | batch 40/1522
[I 2022-11-10 18:03:25] ...trap/engines/engine.py.325:       elapsed: 0:01:14 | left: 0:46:46
[I 2022-11-10 18:03:25] ...trap/engines/engine.py.326:       process: 3.83842 | load: 2.64215
[I 2022-11-10 18:03:41] ...trap/engines/engine.py.321: eval: epoch 59 | batch 50/1522
[I 2022-11-10 18:03:41] ...trap/engines/engine.py.325:       elapsed: 0:01:30 | left: 0:40:43
[I 2022-11-10 18:03:41] ...trap/engines/engine.py.326:       process: 1.20910 | load: 0.00012
[I 2022-11-10 18:03:58] ...trap/engines/engine.py.321: eval: epoch 59 | batch 60/1522
[I 2022-11-10 18:03:58] ...trap/engines/engine.py.325:       elapsed: 0:01:47 | left: 0:44:01
[I 2022-11-10 18:03:58] ...trap/engines/engine.py.326:       process: 3.12216 | load: 1.94155
[I 2022-11-10 18:04:14] ...trap/engines/engine.py.321: eval: epoch 59 | batch 70/1522
[I 2022-11-10 18:04:14] ...trap/engines/engine.py.325:       elapsed: 0:02:03 | left: 0:37:56
[I 2022-11-10 18:04:14] ...trap/engines/engine.py.326:       process: 1.15780 | load: 0.00012
[I 2022-11-10 18:04:31] ...trap/engines/engine.py.321: eval: epoch 59 | batch 80/1522
[I 2022-11-10 18:04:31] ...trap/engines/engine.py.325:       elapsed: 0:02:20 | left: 0:42:18
[I 2022-11-10 18:04:31] ...trap/engines/engine.py.326:       process: 3.04645 | load: 1.88117
[I 2022-11-10 18:04:47] ...trap/engines/engine.py.321: eval: epoch 59 | batch 90/1522
[I 2022-11-10 18:04:47] ...trap/engines/engine.py.325:       elapsed: 0:02:36 | left: 0:38:32
[I 2022-11-10 18:04:47] ...trap/engines/engine.py.326:       process: 1.17259 | load: 0.00012
[I 2022-11-10 18:05:06] ...trap/engines/engine.py.321: eval: epoch 59 | batch 100/1522
[I 2022-11-10 18:05:06] ...trap/engines/engine.py.325:       elapsed: 0:02:55 | left: 0:46:48
[I 2022-11-10 18:05:06] ...trap/engines/engine.py.326:       process: 3.47362 | load: 2.30136
[I 2022-11-10 18:05:20] ...trap/engines/engine.py.321: eval: epoch 59 | batch 110/1522
[I 2022-11-10 18:05:20] ...trap/engines/engine.py.325:       elapsed: 0:03:09 | left: 0:35:07
[I 2022-11-10 18:05:20] ...trap/engines/engine.py.326:       process: 1.18156 | load: 0.00013
[I 2022-11-10 18:05:36] ...trap/engines/engine.py.321: eval: epoch 59 | batch 120/1522
[I 2022-11-10 18:05:36] ...trap/engines/engine.py.325:       elapsed: 0:03:26 | left: 0:41:18
[I 2022-11-10 18:05:36] ...trap/engines/engine.py.326:       process: 3.02954 | load: 1.86138
[I 2022-11-10 18:05:53] ...trap/engines/engine.py.321: eval: epoch 59 | batch 130/1522
[I 2022-11-10 18:05:53] ...trap/engines/engine.py.325:       elapsed: 0:03:42 | left: 0:38:38
[I 2022-11-10 18:05:53] ...trap/engines/engine.py.326:       process: 1.17986 | load: 0.00012
[I 2022-11-10 18:06:10] ...trap/engines/engine.py.321: eval: epoch 59 | batch 140/1522
[I 2022-11-10 18:06:10] ...trap/engines/engine.py.325:       elapsed: 0:03:59 | left: 0:41:35
[I 2022-11-10 18:06:10] ...trap/engines/engine.py.326:       process: 2.99051 | load: 1.82668
[I 2022-11-10 18:06:27] ...trap/engines/engine.py.321: eval: epoch 59 | batch 150/1522
[I 2022-11-10 18:06:27] ...trap/engines/engine.py.325:       elapsed: 0:04:16 | left: 0:38:45
[I 2022-11-10 18:06:27] ...trap/engines/engine.py.326:       process: 1.16200 | load: 0.00013
[I 2022-11-10 18:06:45] ...trap/engines/engine.py.321: eval: epoch 59 | batch 160/1522
[I 2022-11-10 18:06:45] ...trap/engines/engine.py.325:       elapsed: 0:04:35 | left: 0:44:45
[I 2022-11-10 18:06:45] ...trap/engines/engine.py.326:       process: 3.50696 | load: 2.30643
[I 2022-11-10 18:07:01] ...trap/engines/engine.py.321: eval: epoch 59 | batch 170/1522
[I 2022-11-10 18:07:01] ...trap/engines/engine.py.325:       elapsed: 0:04:50 | left: 0:36:35
[I 2022-11-10 18:07:01] ...trap/engines/engine.py.326:       process: 1.16868 | load: 0.00012
[I 2022-11-10 18:07:20] ...trap/engines/engine.py.321: eval: epoch 59 | batch 180/1522
[I 2022-11-10 18:07:20] ...trap/engines/engine.py.325:       elapsed: 0:05:09 | left: 0:42:58
[I 2022-11-10 18:07:20] ...trap/engines/engine.py.326:       process: 3.29511 | load: 2.12390
[I 2022-11-10 18:07:35] ...trap/engines/engine.py.321: eval: epoch 59 | batch 190/1522
[I 2022-11-10 18:07:35] ...trap/engines/engine.py.325:       elapsed: 0:05:25 | left: 0:33:50
[I 2022-11-10 18:07:35] ...trap/engines/engine.py.326:       process: 1.17610 | load: 0.00012
[I 2022-11-10 18:07:55] ...trap/engines/engine.py.321: eval: epoch 59 | batch 200/1522
[I 2022-11-10 18:07:55] ...trap/engines/engine.py.325:       elapsed: 0:05:45 | left: 0:47:54
[I 2022-11-10 18:07:55] ...trap/engines/engine.py.326:       process: 4.66700 | load: 3.51192
[I 2022-11-10 18:08:13] ...trap/engines/engine.py.321: eval: epoch 59 | batch 210/1522
[I 2022-11-10 18:08:13] ...trap/engines/engine.py.325:       elapsed: 0:06:02 | left: 0:39:56
[I 2022-11-10 18:08:13] ...trap/engines/engine.py.326:       process: 1.12702 | load: 0.00012
[I 2022-11-10 18:08:32] ...trap/engines/engine.py.321: eval: epoch 59 | batch 220/1522
[I 2022-11-10 18:08:32] ...trap/engines/engine.py.325:       elapsed: 0:06:21 | left: 0:44:04
[I 2022-11-10 18:08:32] ...trap/engines/engine.py.326:       process: 3.72821 | load: 2.55352
[I 2022-11-10 18:08:50] ...trap/engines/engine.py.321: eval: epoch 59 | batch 230/1522
[I 2022-11-10 18:08:50] ...trap/engines/engine.py.325:       elapsed: 0:06:39 | left: 0:39:52
[I 2022-11-10 18:08:50] ...trap/engines/engine.py.326:       process: 1.16271 | load: 0.00012
[I 2022-11-10 18:09:08] ...trap/engines/engine.py.321: eval: epoch 59 | batch 240/1522
[I 2022-11-10 18:09:08] ...trap/engines/engine.py.325:       elapsed: 0:06:57 | left: 0:41:15
[I 2022-11-10 18:09:08] ...trap/engines/engine.py.326:       process: 3.42943 | load: 2.26830
[I 2022-11-10 18:09:25] ...trap/engines/engine.py.321: eval: epoch 59 | batch 250/1522
[I 2022-11-10 18:09:25] ...trap/engines/engine.py.325:       elapsed: 0:07:14 | left: 0:36:08
[I 2022-11-10 18:09:25] ...trap/engines/engine.py.326:       process: 1.19995 | load: 0.00013
[I 2022-11-10 18:09:44] ...trap/engines/engine.py.321: eval: epoch 59 | batch 260/1522
[I 2022-11-10 18:09:44] ...trap/engines/engine.py.325:       elapsed: 0:07:33 | left: 0:41:14
[I 2022-11-10 18:09:44] ...trap/engines/engine.py.326:       process: 3.41200 | load: 2.21962
[I 2022-11-10 18:09:59] ...trap/engines/engine.py.321: eval: epoch 59 | batch 270/1522
[I 2022-11-10 18:09:59] ...trap/engines/engine.py.325:       elapsed: 0:07:48 | left: 0:32:14
[I 2022-11-10 18:09:59] ...trap/engines/engine.py.326:       process: 1.16795 | load: 0.00013
[I 2022-11-10 18:10:17] ...trap/engines/engine.py.321: eval: epoch 59 | batch 280/1522
[I 2022-11-10 18:10:17] ...trap/engines/engine.py.325:       elapsed: 0:08:07 | left: 0:41:38
[I 2022-11-10 18:10:17] ...trap/engines/engine.py.326:       process: 3.84015 | load: 2.65238
[I 2022-11-10 18:10:34] ...trap/engines/engine.py.321: eval: epoch 59 | batch 290/1522
[I 2022-11-10 18:10:34] ...trap/engines/engine.py.325:       elapsed: 0:08:23 | left: 0:34:16
[I 2022-11-10 18:10:34] ...trap/engines/engine.py.326:       process: 1.14512 | load: 0.00012
[I 2022-11-10 18:10:52] ...trap/engines/engine.py.321: eval: epoch 59 | batch 300/1522
[I 2022-11-10 18:10:52] ...trap/engines/engine.py.325:       elapsed: 0:08:41 | left: 0:38:38
[I 2022-11-10 18:10:52] ...trap/engines/engine.py.326:       process: 3.30469 | load: 2.13706
[I 2022-11-10 18:11:07] ...trap/engines/engine.py.321: eval: epoch 59 | batch 310/1522
[I 2022-11-10 18:11:07] ...trap/engines/engine.py.325:       elapsed: 0:08:57 | left: 0:31:55
[I 2022-11-10 18:11:07] ...trap/engines/engine.py.326:       process: 1.20289 | load: 0.00013
[I 2022-11-10 18:11:26] ...trap/engines/engine.py.321: eval: epoch 59 | batch 320/1522
[I 2022-11-10 18:11:26] ...trap/engines/engine.py.325:       elapsed: 0:09:15 | left: 0:35:55
[I 2022-11-10 18:11:26] ...trap/engines/engine.py.326:       process: 2.34135 | load: 1.16052
[I 2022-11-10 18:11:42] ...trap/engines/engine.py.321: eval: epoch 59 | batch 330/1522
[I 2022-11-10 18:11:42] ...trap/engines/engine.py.325:       elapsed: 0:09:31 | left: 0:31:57
[I 2022-11-10 18:11:42] ...trap/engines/engine.py.326:       process: 1.18950 | load: 0.00013
[I 2022-11-10 18:12:02] ...trap/engines/engine.py.321: eval: epoch 59 | batch 340/1522
[I 2022-11-10 18:12:02] ...trap/engines/engine.py.325:       elapsed: 0:09:51 | left: 0:42:06
[I 2022-11-10 18:12:02] ...trap/engines/engine.py.326:       process: 3.90564 | load: 2.76236
[I 2022-11-10 18:12:20] ...trap/engines/engine.py.321: eval: epoch 59 | batch 350/1522
[I 2022-11-10 18:12:20] ...trap/engines/engine.py.325:       elapsed: 0:10:09 | left: 0:35:59
[I 2022-11-10 18:12:20] ...trap/engines/engine.py.326:       process: 1.15050 | load: 0.00013
[I 2022-11-10 18:12:38] ...trap/engines/engine.py.321: eval: epoch 59 | batch 360/1522
[I 2022-11-10 18:12:38] ...trap/engines/engine.py.325:       elapsed: 0:10:27 | left: 0:37:34
[I 2022-11-10 18:12:38] ...trap/engines/engine.py.326:       process: 3.38138 | load: 2.23109
[I 2022-11-10 18:12:56] ...trap/engines/engine.py.321: eval: epoch 59 | batch 370/1522
[I 2022-11-10 18:12:56] ...trap/engines/engine.py.325:       elapsed: 0:10:45 | left: 0:34:59
[I 2022-11-10 18:12:56] ...trap/engines/engine.py.326:       process: 1.13643 | load: 0.00013
[I 2022-11-10 18:13:16] ...trap/engines/engine.py.321: eval: epoch 59 | batch 380/1522
[I 2022-11-10 18:13:16] ...trap/engines/engine.py.325:       elapsed: 0:11:05 | left: 0:41:34
[I 2022-11-10 18:13:16] ...trap/engines/engine.py.326:       process: 4.21131 | load: 3.07546
[I 2022-11-10 18:13:35] ...trap/engines/engine.py.321: eval: epoch 59 | batch 390/1522
[I 2022-11-10 18:13:35] ...trap/engines/engine.py.325:       elapsed: 0:11:24 | left: 0:36:04
[I 2022-11-10 18:13:35] ...trap/engines/engine.py.326:       process: 1.14286 | load: 0.00012
[I 2022-11-10 18:13:54] ...trap/engines/engine.py.321: eval: epoch 59 | batch 400/1522
[I 2022-11-10 18:13:54] ...trap/engines/engine.py.325:       elapsed: 0:11:44 | left: 0:38:40
[I 2022-11-10 18:13:54] ...trap/engines/engine.py.326:       process: 3.87941 | load: 2.70311
[I 2022-11-10 18:14:12] ...trap/engines/engine.py.321: eval: epoch 59 | batch 410/1522
[I 2022-11-10 18:14:12] ...trap/engines/engine.py.325:       elapsed: 0:12:01 | left: 0:33:06
[I 2022-11-10 18:14:12] ...trap/engines/engine.py.326:       process: 1.14323 | load: 0.00007
[I 2022-11-10 18:14:30] ...trap/engines/engine.py.321: eval: epoch 59 | batch 420/1522
[I 2022-11-10 18:14:30] ...trap/engines/engine.py.325:       elapsed: 0:12:20 | left: 0:33:33
[I 2022-11-10 18:14:30] ...trap/engines/engine.py.326:       process: 1.89012 | load: 0.74031
[I 2022-11-10 18:14:45] ...trap/engines/engine.py.321: eval: epoch 59 | batch 430/1522
[I 2022-11-10 18:14:45] ...trap/engines/engine.py.325:       elapsed: 0:12:35 | left: 0:27:09
[I 2022-11-10 18:14:45] ...trap/engines/engine.py.326:       process: 1.14261 | load: 0.00007
[I 2022-11-10 18:15:02] ...trap/engines/engine.py.321: eval: epoch 59 | batch 440/1522
[I 2022-11-10 18:15:02] ...trap/engines/engine.py.325:       elapsed: 0:12:52 | left: 0:31:37
[I 2022-11-10 18:15:02] ...trap/engines/engine.py.326:       process: 2.86963 | load: 1.72785
[I 2022-11-10 18:15:18] ...trap/engines/engine.py.321: eval: epoch 59 | batch 450/1522
[I 2022-11-10 18:15:18] ...trap/engines/engine.py.325:       elapsed: 0:13:07 | left: 0:30:14
[I 2022-11-10 18:15:18] ...trap/engines/engine.py.326:       process: 1.15197 | load: 0.00007
[I 2022-11-10 18:15:34] ...trap/engines/engine.py.321: eval: epoch 59 | batch 460/1522
[I 2022-11-10 18:15:34] ...trap/engines/engine.py.325:       elapsed: 0:13:23 | left: 0:28:23
[I 2022-11-10 18:15:34] ...trap/engines/engine.py.326:       process: 2.00240 | load: 0.86401
[I 2022-11-10 18:15:51] ...trap/engines/engine.py.321: eval: epoch 59 | batch 470/1522
[I 2022-11-10 18:15:51] ...trap/engines/engine.py.325:       elapsed: 0:13:40 | left: 0:29:33
[I 2022-11-10 18:15:51] ...trap/engines/engine.py.326:       process: 1.15250 | load: 0.00012
[I 2022-11-10 18:16:09] ...trap/engines/engine.py.321: eval: epoch 59 | batch 480/1522
[I 2022-11-10 18:16:09] ...trap/engines/engine.py.325:       elapsed: 0:13:58 | left: 0:31:08
[I 2022-11-10 18:16:09] ...trap/engines/engine.py.326:       process: 2.91634 | load: 1.75958
[I 2022-11-10 18:16:23] ...trap/engines/engine.py.321: eval: epoch 59 | batch 490/1522
[I 2022-11-10 18:16:23] ...trap/engines/engine.py.325:       elapsed: 0:14:13 | left: 0:25:25
[I 2022-11-10 18:16:23] ...trap/engines/engine.py.326:       process: 1.14553 | load: 0.00014
[I 2022-11-10 18:16:40] ...trap/engines/engine.py.321: eval: epoch 59 | batch 500/1522
[I 2022-11-10 18:16:40] ...trap/engines/engine.py.325:       elapsed: 0:14:29 | left: 0:27:53
[I 2022-11-10 18:16:40] ...trap/engines/engine.py.326:       process: 2.43339 | load: 1.36973
[I 2022-11-10 18:16:56] ...trap/engines/engine.py.321: eval: epoch 59 | batch 510/1522
[I 2022-11-10 18:16:56] ...trap/engines/engine.py.325:       elapsed: 0:14:45 | left: 0:26:51
[I 2022-11-10 18:16:56] ...trap/engines/engine.py.326:       process: 1.07157 | load: 0.00012
[I 2022-11-10 18:17:12] ...trap/engines/engine.py.321: eval: epoch 59 | batch 520/1522
[I 2022-11-10 18:17:12] ...trap/engines/engine.py.325:       elapsed: 0:15:01 | left: 0:29:19
[I 2022-11-10 18:17:12] ...trap/engines/engine.py.326:       process: 3.11937 | load: 2.04931
[I 2022-11-10 18:17:27] ...trap/engines/engine.py.321: eval: epoch 59 | batch 530/1522
[I 2022-11-10 18:17:27] ...trap/engines/engine.py.325:       elapsed: 0:15:16 | left: 0:26:43
[I 2022-11-10 18:17:27] ...trap/engines/engine.py.326:       process: 1.06749 | load: 0.00013
[I 2022-11-10 18:17:44] ...trap/engines/engine.py.321: eval: epoch 59 | batch 540/1522
[I 2022-11-10 18:17:44] ...trap/engines/engine.py.325:       elapsed: 0:15:33 | left: 0:30:52
[I 2022-11-10 18:17:44] ...trap/engines/engine.py.326:       process: 3.84811 | load: 2.76692
[I 2022-11-10 18:17:58] ...trap/engines/engine.py.321: eval: epoch 59 | batch 550/1522
[I 2022-11-10 18:17:58] ...trap/engines/engine.py.325:       elapsed: 0:15:48 | left: 0:23:50
[I 2022-11-10 18:17:58] ...trap/engines/engine.py.326:       process: 1.07233 | load: 0.00007
[I 2022-11-10 18:18:16] ...trap/engines/engine.py.321: eval: epoch 59 | batch 560/1522
[I 2022-11-10 18:18:16] ...trap/engines/engine.py.325:       elapsed: 0:16:05 | left: 0:29:32
[I 2022-11-10 18:18:16] ...trap/engines/engine.py.326:       process: 3.56940 | load: 2.50524
[I 2022-11-10 18:18:30] ...trap/engines/engine.py.321: eval: epoch 59 | batch 570/1522
[I 2022-11-10 18:18:30] ...trap/engines/engine.py.325:       elapsed: 0:16:19 | left: 0:23:44
[I 2022-11-10 18:18:30] ...trap/engines/engine.py.326:       process: 1.06704 | load: 0.00007
[I 2022-11-10 18:18:46] ...trap/engines/engine.py.321: eval: epoch 59 | batch 580/1522
[I 2022-11-10 18:18:46] ...trap/engines/engine.py.325:       elapsed: 0:16:35 | left: 0:25:06
[I 2022-11-10 18:18:46] ...trap/engines/engine.py.326:       process: 2.31538 | load: 1.25363
[I 2022-11-10 18:19:03] ...trap/engines/engine.py.321: eval: epoch 59 | batch 590/1522
[I 2022-11-10 18:19:03] ...trap/engines/engine.py.325:       elapsed: 0:16:52 | left: 0:27:29
[I 2022-11-10 18:19:03] ...trap/engines/engine.py.326:       process: 1.06948 | load: 0.00013
[I 2022-11-10 18:19:18] ...trap/engines/engine.py.321: eval: epoch 59 | batch 600/1522
[I 2022-11-10 18:19:18] ...trap/engines/engine.py.325:       elapsed: 0:17:07 | left: 0:22:03
[I 2022-11-10 18:19:18] ...trap/engines/engine.py.326:       process: 1.07223 | load: 0.00015
[I 2022-11-10 18:19:34] ...trap/engines/engine.py.321: eval: epoch 59 | batch 610/1522
[I 2022-11-10 18:19:34] ...trap/engines/engine.py.325:       elapsed: 0:17:23 | left: 0:22:16
[I 2022-11-10 18:19:34] ...trap/engines/engine.py.326:       process: 1.06932 | load: 0.00007
[I 2022-11-10 18:19:48] ...trap/engines/engine.py.321: eval: epoch 59 | batch 620/1522
[I 2022-11-10 18:19:48] ...trap/engines/engine.py.325:       elapsed: 0:17:38 | left: 0:22:47
[I 2022-11-10 18:19:48] ...trap/engines/engine.py.326:       process: 2.01471 | load: 0.94559
[I 2022-11-10 18:20:04] ...trap/engines/engine.py.321: eval: epoch 59 | batch 630/1522
[I 2022-11-10 18:20:04] ...trap/engines/engine.py.325:       elapsed: 0:17:53 | left: 0:21:29
[I 2022-11-10 18:20:04] ...trap/engines/engine.py.326:       process: 1.06842 | load: 0.00007
[I 2022-11-10 18:20:20] ...trap/engines/engine.py.321: eval: epoch 59 | batch 640/1522
[I 2022-11-10 18:20:20] ...trap/engines/engine.py.325:       elapsed: 0:18:09 | left: 0:23:49
[I 2022-11-10 18:20:20] ...trap/engines/engine.py.326:       process: 2.46582 | load: 1.40378
[I 2022-11-10 18:20:36] ...trap/engines/engine.py.321: eval: epoch 59 | batch 650/1522
[I 2022-11-10 18:20:36] ...trap/engines/engine.py.325:       elapsed: 0:18:25 | left: 0:23:12
[I 2022-11-10 18:20:36] ...trap/engines/engine.py.326:       process: 1.06563 | load: 0.00007
[I 2022-11-10 18:20:50] ...trap/engines/engine.py.321: eval: epoch 59 | batch 660/1522
[I 2022-11-10 18:20:50] ...trap/engines/engine.py.325:       elapsed: 0:18:40 | left: 0:21:14
[I 2022-11-10 18:20:50] ...trap/engines/engine.py.326:       process: 1.75626 | load: 0.68142
[I 2022-11-10 18:21:06] ...trap/engines/engine.py.321: eval: epoch 59 | batch 670/1522
[I 2022-11-10 18:21:06] ...trap/engines/engine.py.325:       elapsed: 0:18:56 | left: 0:22:38
[I 2022-11-10 18:21:06] ...trap/engines/engine.py.326:       process: 1.07033 | load: 0.00007
[I 2022-11-10 18:21:21] ...trap/engines/engine.py.321: eval: epoch 59 | batch 680/1522
[I 2022-11-10 18:21:21] ...trap/engines/engine.py.325:       elapsed: 0:19:10 | left: 0:19:41
[I 2022-11-10 18:21:21] ...trap/engines/engine.py.326:       process: 1.27523 | load: 0.20797
[I 2022-11-10 18:21:36] ...trap/engines/engine.py.321: eval: epoch 59 | batch 690/1522
[I 2022-11-10 18:21:36] ...trap/engines/engine.py.325:       elapsed: 0:19:26 | left: 0:20:56
[I 2022-11-10 18:21:36] ...trap/engines/engine.py.326:       process: 1.07121 | load: 0.00008
[I 2022-11-10 18:21:52] ...trap/engines/engine.py.321: eval: epoch 59 | batch 700/1522
[I 2022-11-10 18:21:52] ...trap/engines/engine.py.325:       elapsed: 0:19:41 | left: 0:19:07
[I 2022-11-10 18:21:52] ...trap/engines/engine.py.326:       process: 1.22653 | load: 0.15643
[I 2022-11-10 18:22:08] ...trap/engines/engine.py.321: eval: epoch 59 | batch 710/1522
[I 2022-11-10 18:22:08] ...trap/engines/engine.py.325:       elapsed: 0:19:57 | left: 0:20:04
[I 2022-11-10 18:22:08] ...trap/engines/engine.py.326:       process: 1.06864 | load: 0.00007
[I 2022-11-10 18:22:24] ...trap/engines/engine.py.321: eval: epoch 59 | batch 720/1522
[I 2022-11-10 18:22:24] ...trap/engines/engine.py.325:       elapsed: 0:20:13 | left: 0:19:44
[I 2022-11-10 18:22:24] ...trap/engines/engine.py.326:       process: 1.07820 | load: 0.00016
[I 2022-11-10 18:22:41] ...trap/engines/engine.py.321: eval: epoch 59 | batch 730/1522
[I 2022-11-10 18:22:41] ...trap/engines/engine.py.325:       elapsed: 0:20:30 | left: 0:20:51
[I 2022-11-10 18:22:41] ...trap/engines/engine.py.326:       process: 1.06850 | load: 0.00007
[I 2022-11-10 18:22:55] ...trap/engines/engine.py.321: eval: epoch 59 | batch 740/1522
[I 2022-11-10 18:22:55] ...trap/engines/engine.py.325:       elapsed: 0:20:44 | left: 0:18:34
[I 2022-11-10 18:22:55] ...trap/engines/engine.py.326:       process: 1.07056 | load: 0.00015
[I 2022-11-10 18:23:12] ...trap/engines/engine.py.321: eval: epoch 59 | batch 750/1522
[I 2022-11-10 18:23:12] ...trap/engines/engine.py.325:       elapsed: 0:21:01 | left: 0:22:05
[I 2022-11-10 18:23:12] ...trap/engines/engine.py.326:       process: 1.06880 | load: 0.00013
[I 2022-11-10 18:23:26] ...trap/engines/engine.py.321: eval: epoch 59 | batch 760/1522
[I 2022-11-10 18:23:26] ...trap/engines/engine.py.325:       elapsed: 0:21:15 | left: 0:17:55
[I 2022-11-10 18:23:26] ...trap/engines/engine.py.326:       process: 1.07448 | load: 0.00015
[I 2022-11-10 18:23:43] ...trap/engines/engine.py.321: eval: epoch 59 | batch 770/1522
[I 2022-11-10 18:23:43] ...trap/engines/engine.py.325:       elapsed: 0:21:33 | left: 0:20:32
[I 2022-11-10 18:23:43] ...trap/engines/engine.py.326:       process: 1.07255 | load: 0.00007
[I 2022-11-10 18:23:59] ...trap/engines/engine.py.321: eval: epoch 59 | batch 780/1522
[I 2022-11-10 18:23:59] ...trap/engines/engine.py.325:       elapsed: 0:21:48 | left: 0:17:55
[I 2022-11-10 18:23:59] ...trap/engines/engine.py.326:       process: 1.07509 | load: 0.00013
[I 2022-11-10 18:24:15] ...trap/engines/engine.py.321: eval: epoch 59 | batch 790/1522
[I 2022-11-10 18:24:15] ...trap/engines/engine.py.325:       elapsed: 0:22:04 | left: 0:18:35
[I 2022-11-10 18:24:15] ...trap/engines/engine.py.326:       process: 1.06999 | load: 0.00007
[I 2022-11-10 18:24:30] ...trap/engines/engine.py.321: eval: epoch 59 | batch 800/1522
[I 2022-11-10 18:24:30] ...trap/engines/engine.py.325:       elapsed: 0:22:19 | left: 0:16:26
[I 2022-11-10 18:24:30] ...trap/engines/engine.py.326:       process: 1.06954 | load: 0.00013
[I 2022-11-10 18:24:46] ...trap/engines/engine.py.321: eval: epoch 59 | batch 810/1522
[I 2022-11-10 18:24:46] ...trap/engines/engine.py.325:       elapsed: 0:22:35 | left: 0:19:30
[I 2022-11-10 18:24:46] ...trap/engines/engine.py.326:       process: 1.06744 | load: 0.00007
[I 2022-11-10 18:25:01] ...trap/engines/engine.py.321: eval: epoch 59 | batch 820/1522
[I 2022-11-10 18:25:01] ...trap/engines/engine.py.325:       elapsed: 0:22:50 | left: 0:16:25
[I 2022-11-10 18:25:01] ...trap/engines/engine.py.326:       process: 1.07071 | load: 0.00012
[I 2022-11-10 18:25:17] ...trap/engines/engine.py.321: eval: epoch 59 | batch 830/1522
[I 2022-11-10 18:25:17] ...trap/engines/engine.py.325:       elapsed: 0:23:06 | left: 0:18:48
[I 2022-11-10 18:25:17] ...trap/engines/engine.py.326:       process: 1.07159 | load: 0.00007
[I 2022-11-10 18:25:31] ...trap/engines/engine.py.321: eval: epoch 59 | batch 840/1522
[I 2022-11-10 18:25:31] ...trap/engines/engine.py.325:       elapsed: 0:23:20 | left: 0:15:14
[I 2022-11-10 18:25:31] ...trap/engines/engine.py.326:       process: 1.07215 | load: 0.00013
[I 2022-11-10 18:25:47] ...trap/engines/engine.py.321: eval: epoch 59 | batch 850/1522
[I 2022-11-10 18:25:47] ...trap/engines/engine.py.325:       elapsed: 0:23:36 | left: 0:16:44
[I 2022-11-10 18:25:47] ...trap/engines/engine.py.326:       process: 1.07103 | load: 0.00007
[I 2022-11-10 18:26:01] ...trap/engines/engine.py.321: eval: epoch 59 | batch 860/1522
[I 2022-11-10 18:26:01] ...trap/engines/engine.py.325:       elapsed: 0:23:50 | left: 0:15:14
[I 2022-11-10 18:26:01] ...trap/engines/engine.py.326:       process: 1.07813 | load: 0.00013
[I 2022-11-10 18:26:19] ...trap/engines/engine.py.321: eval: epoch 59 | batch 870/1522
[I 2022-11-10 18:26:19] ...trap/engines/engine.py.325:       elapsed: 0:24:08 | left: 0:18:30
[I 2022-11-10 18:26:19] ...trap/engines/engine.py.326:       process: 1.07186 | load: 0.00008
[I 2022-11-10 18:26:32] ...trap/engines/engine.py.321: eval: epoch 59 | batch 880/1522
[I 2022-11-10 18:26:32] ...trap/engines/engine.py.325:       elapsed: 0:24:21 | left: 0:14:33
[I 2022-11-10 18:26:32] ...trap/engines/engine.py.326:       process: 1.07471 | load: 0.00018
[I 2022-11-10 18:26:49] ...trap/engines/engine.py.321: eval: epoch 59 | batch 890/1522
[I 2022-11-10 18:26:49] ...trap/engines/engine.py.325:       elapsed: 0:24:39 | left: 0:17:56
[I 2022-11-10 18:26:49] ...trap/engines/engine.py.326:       process: 1.06544 | load: 0.00012
[I 2022-11-10 18:27:04] ...trap/engines/engine.py.321: eval: epoch 59 | batch 900/1522
[I 2022-11-10 18:27:04] ...trap/engines/engine.py.325:       elapsed: 0:24:54 | left: 0:14:56
[I 2022-11-10 18:27:04] ...trap/engines/engine.py.326:       process: 1.07331 | load: 0.00013
[I 2022-11-10 18:27:20] ...trap/engines/engine.py.321: eval: epoch 59 | batch 910/1522
[I 2022-11-10 18:27:20] ...trap/engines/engine.py.325:       elapsed: 0:25:09 | left: 0:14:52
[I 2022-11-10 18:27:20] ...trap/engines/engine.py.326:       process: 1.06499 | load: 0.00007
[I 2022-11-10 18:27:34] ...trap/engines/engine.py.321: eval: epoch 59 | batch 920/1522
[I 2022-11-10 18:27:34] ...trap/engines/engine.py.325:       elapsed: 0:25:23 | left: 0:13:59
[I 2022-11-10 18:27:34] ...trap/engines/engine.py.326:       process: 1.07244 | load: 0.00013
[I 2022-11-10 18:27:51] ...trap/engines/engine.py.321: eval: epoch 59 | batch 930/1522
[I 2022-11-10 18:27:51] ...trap/engines/engine.py.325:       elapsed: 0:25:40 | left: 0:16:11
[I 2022-11-10 18:27:51] ...trap/engines/engine.py.326:       process: 1.07749 | load: 0.00007
[I 2022-11-10 18:28:06] ...trap/engines/engine.py.321: eval: epoch 59 | batch 940/1522
[I 2022-11-10 18:28:06] ...trap/engines/engine.py.325:       elapsed: 0:25:55 | left: 0:13:49
[I 2022-11-10 18:28:06] ...trap/engines/engine.py.326:       process: 1.07150 | load: 0.00007
[I 2022-11-10 18:28:22] ...trap/engines/engine.py.321: eval: epoch 59 | batch 950/1522
[I 2022-11-10 18:28:22] ...trap/engines/engine.py.325:       elapsed: 0:26:11 | left: 0:14:29
[I 2022-11-10 18:28:22] ...trap/engines/engine.py.326:       process: 1.08214 | load: 0.00007
[I 2022-11-10 18:28:38] ...trap/engines/engine.py.321: eval: epoch 59 | batch 960/1522
[I 2022-11-10 18:28:38] ...trap/engines/engine.py.325:       elapsed: 0:26:27 | left: 0:14:48
[I 2022-11-10 18:28:38] ...trap/engines/engine.py.326:       process: 1.07394 | load: 0.00007
[I 2022-11-10 18:28:54] ...trap/engines/engine.py.321: eval: epoch 59 | batch 970/1522
[I 2022-11-10 18:28:54] ...trap/engines/engine.py.325:       elapsed: 0:26:43 | left: 0:14:43
[I 2022-11-10 18:28:54] ...trap/engines/engine.py.326:       process: 1.06564 | load: 0.00013
[I 2022-11-10 18:29:11] ...trap/engines/engine.py.321: eval: epoch 59 | batch 980/1522
[I 2022-11-10 18:29:11] ...trap/engines/engine.py.325:       elapsed: 0:27:00 | left: 0:15:38
[I 2022-11-10 18:29:11] ...trap/engines/engine.py.326:       process: 1.06528 | load: 0.00014
[I 2022-11-10 18:29:26] ...trap/engines/engine.py.321: eval: epoch 59 | batch 990/1522
[I 2022-11-10 18:29:26] ...trap/engines/engine.py.325:       elapsed: 0:27:15 | left: 0:12:06
[I 2022-11-10 18:29:26] ...trap/engines/engine.py.326:       process: 1.06682 | load: 0.00012
[I 2022-11-10 18:29:43] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1000/1522
[I 2022-11-10 18:29:43] ...trap/engines/engine.py.325:       elapsed: 0:27:32 | left: 0:13:28
[I 2022-11-10 18:29:43] ...trap/engines/engine.py.326:       process: 1.06872 | load: 0.00013
[I 2022-11-10 18:29:58] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1010/1522
[I 2022-11-10 18:29:58] ...trap/engines/engine.py.325:       elapsed: 0:27:48 | left: 0:12:37
[I 2022-11-10 18:29:58] ...trap/engines/engine.py.326:       process: 1.06286 | load: 0.00013
[I 2022-11-10 18:30:15] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1020/1522
[I 2022-11-10 18:30:15] ...trap/engines/engine.py.325:       elapsed: 0:28:04 | left: 0:13:40
[I 2022-11-10 18:30:15] ...trap/engines/engine.py.326:       process: 1.06609 | load: 0.00012
[I 2022-11-10 18:30:28] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1030/1522
[I 2022-11-10 18:30:28] ...trap/engines/engine.py.325:       elapsed: 0:28:18 | left: 0:11:04
[I 2022-11-10 18:30:28] ...trap/engines/engine.py.326:       process: 1.07645 | load: 0.00013
[I 2022-11-10 18:30:45] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1040/1522
[I 2022-11-10 18:30:45] ...trap/engines/engine.py.325:       elapsed: 0:28:35 | left: 0:12:41
[I 2022-11-10 18:30:45] ...trap/engines/engine.py.326:       process: 1.06936 | load: 0.00013
[I 2022-11-10 18:31:00] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1050/1522
[I 2022-11-10 18:31:00] ...trap/engines/engine.py.325:       elapsed: 0:28:49 | left: 0:10:55
[I 2022-11-10 18:31:00] ...trap/engines/engine.py.326:       process: 1.06973 | load: 0.00013
[I 2022-11-10 18:31:16] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1060/1522
[I 2022-11-10 18:31:16] ...trap/engines/engine.py.325:       elapsed: 0:29:05 | left: 0:11:59
[I 2022-11-10 18:31:16] ...trap/engines/engine.py.326:       process: 1.08020 | load: 0.00007
[I 2022-11-10 18:31:32] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1070/1522
[I 2022-11-10 18:31:32] ...trap/engines/engine.py.325:       elapsed: 0:29:21 | left: 0:11:19
[I 2022-11-10 18:31:32] ...trap/engines/engine.py.326:       process: 1.06815 | load: 0.00013
[I 2022-11-10 18:31:48] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1080/1522
[I 2022-11-10 18:31:48] ...trap/engines/engine.py.325:       elapsed: 0:29:37 | left: 0:11:58
[I 2022-11-10 18:31:48] ...trap/engines/engine.py.326:       process: 1.06228 | load: 0.00012
[I 2022-11-10 18:32:03] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1090/1522
[I 2022-11-10 18:32:03] ...trap/engines/engine.py.325:       elapsed: 0:29:52 | left: 0:10:28
[I 2022-11-10 18:32:03] ...trap/engines/engine.py.326:       process: 1.06948 | load: 0.00013
[I 2022-11-10 18:32:19] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1100/1522
[I 2022-11-10 18:32:19] ...trap/engines/engine.py.325:       elapsed: 0:30:08 | left: 0:10:45
[I 2022-11-10 18:32:19] ...trap/engines/engine.py.326:       process: 1.06742 | load: 0.00008
[I 2022-11-10 18:32:32] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1110/1522
[I 2022-11-10 18:32:32] ...trap/engines/engine.py.325:       elapsed: 0:30:22 | left: 0:09:01
[I 2022-11-10 18:32:32] ...trap/engines/engine.py.326:       process: 1.07720 | load: 0.00012
[I 2022-11-10 18:32:48] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1120/1522
[I 2022-11-10 18:32:48] ...trap/engines/engine.py.325:       elapsed: 0:30:37 | left: 0:10:04
[I 2022-11-10 18:32:48] ...trap/engines/engine.py.326:       process: 1.07390 | load: 0.00007
[I 2022-11-10 18:33:02] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1130/1522
[I 2022-11-10 18:33:02] ...trap/engines/engine.py.325:       elapsed: 0:30:51 | left: 0:08:58
[I 2022-11-10 18:33:02] ...trap/engines/engine.py.326:       process: 1.08679 | load: 0.00007
[I 2022-11-10 18:33:19] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1140/1522
[I 2022-11-10 18:33:19] ...trap/engines/engine.py.325:       elapsed: 0:31:08 | left: 0:09:49
[I 2022-11-10 18:33:19] ...trap/engines/engine.py.326:       process: 1.06735 | load: 0.00008
[I 2022-11-10 18:33:34] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1150/1522
[I 2022-11-10 18:33:34] ...trap/engines/engine.py.325:       elapsed: 0:31:23 | left: 0:08:51
[I 2022-11-10 18:33:34] ...trap/engines/engine.py.326:       process: 1.05821 | load: 0.00015
[I 2022-11-10 18:33:52] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1160/1522
[I 2022-11-10 18:33:52] ...trap/engines/engine.py.325:       elapsed: 0:31:41 | left: 0:10:31
[I 2022-11-10 18:33:52] ...trap/engines/engine.py.326:       process: 1.06908 | load: 0.00013
[I 2022-11-10 18:34:06] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1170/1522
[I 2022-11-10 18:34:06] ...trap/engines/engine.py.325:       elapsed: 0:31:55 | left: 0:08:04
[I 2022-11-10 18:34:06] ...trap/engines/engine.py.326:       process: 1.06654 | load: 0.00015
[I 2022-11-10 18:34:23] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1180/1522
[I 2022-11-10 18:34:23] ...trap/engines/engine.py.325:       elapsed: 0:32:12 | left: 0:09:37
[I 2022-11-10 18:34:23] ...trap/engines/engine.py.326:       process: 1.06804 | load: 0.00007
[I 2022-11-10 18:34:37] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1190/1522
[I 2022-11-10 18:34:37] ...trap/engines/engine.py.325:       elapsed: 0:32:26 | left: 0:07:50
[I 2022-11-10 18:34:37] ...trap/engines/engine.py.326:       process: 1.08608 | load: 0.00016
[I 2022-11-10 18:34:53] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1200/1522
[I 2022-11-10 18:34:53] ...trap/engines/engine.py.325:       elapsed: 0:32:43 | left: 0:08:08
[I 2022-11-10 18:34:53] ...trap/engines/engine.py.326:       process: 1.06934 | load: 0.00012
[I 2022-11-10 18:35:08] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1210/1522
[I 2022-11-10 18:35:08] ...trap/engines/engine.py.325:       elapsed: 0:32:57 | left: 0:07:07
[I 2022-11-10 18:35:08] ...trap/engines/engine.py.326:       process: 1.08690 | load: 0.00015
[I 2022-11-10 18:35:25] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1220/1522
[I 2022-11-10 18:35:25] ...trap/engines/engine.py.325:       elapsed: 0:33:14 | left: 0:08:46
[I 2022-11-10 18:35:25] ...trap/engines/engine.py.326:       process: 1.08936 | load: 0.00014
[I 2022-11-10 18:35:40] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1230/1522
[I 2022-11-10 18:35:40] ...trap/engines/engine.py.325:       elapsed: 0:33:29 | left: 0:07:02
[I 2022-11-10 18:35:40] ...trap/engines/engine.py.326:       process: 1.06786 | load: 0.00016
[I 2022-11-10 18:35:57] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1240/1522
[I 2022-11-10 18:35:57] ...trap/engines/engine.py.325:       elapsed: 0:33:46 | left: 0:07:22
[I 2022-11-10 18:35:57] ...trap/engines/engine.py.326:       process: 1.06564 | load: 0.00013
[I 2022-11-10 18:36:12] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1250/1522
[I 2022-11-10 18:36:12] ...trap/engines/engine.py.325:       elapsed: 0:34:01 | left: 0:06:39
[I 2022-11-10 18:36:12] ...trap/engines/engine.py.326:       process: 1.06827 | load: 0.00015
[I 2022-11-10 18:36:27] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1260/1522
[I 2022-11-10 18:36:27] ...trap/engines/engine.py.325:       elapsed: 0:34:16 | left: 0:06:25
[I 2022-11-10 18:36:27] ...trap/engines/engine.py.326:       process: 1.07045 | load: 0.00007
[I 2022-11-10 18:36:41] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1270/1522
[I 2022-11-10 18:36:41] ...trap/engines/engine.py.325:       elapsed: 0:34:30 | left: 0:05:49
[I 2022-11-10 18:36:41] ...trap/engines/engine.py.326:       process: 1.07356 | load: 0.00015
[I 2022-11-10 18:36:58] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1280/1522
[I 2022-11-10 18:36:58] ...trap/engines/engine.py.325:       elapsed: 0:34:47 | left: 0:06:19
[I 2022-11-10 18:36:58] ...trap/engines/engine.py.326:       process: 1.06710 | load: 0.00015
[I 2022-11-10 18:37:12] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1290/1522
[I 2022-11-10 18:37:12] ...trap/engines/engine.py.325:       elapsed: 0:35:01 | left: 0:05:46
[I 2022-11-10 18:37:12] ...trap/engines/engine.py.326:       process: 1.82184 | load: 0.75317
[I 2022-11-10 18:37:29] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1300/1522
[I 2022-11-10 18:37:29] ...trap/engines/engine.py.325:       elapsed: 0:35:18 | left: 0:06:08
[I 2022-11-10 18:37:29] ...trap/engines/engine.py.326:       process: 1.11093 | load: 0.00013
[I 2022-11-10 18:37:44] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1310/1522
[I 2022-11-10 18:37:44] ...trap/engines/engine.py.325:       elapsed: 0:35:34 | left: 0:05:17
[I 2022-11-10 18:37:44] ...trap/engines/engine.py.326:       process: 1.05518 | load: 0.00014
[I 2022-11-10 18:38:01] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1320/1522
[I 2022-11-10 18:38:01] ...trap/engines/engine.py.325:       elapsed: 0:35:50 | left: 0:05:17
[I 2022-11-10 18:38:01] ...trap/engines/engine.py.326:       process: 1.06550 | load: 0.00012
[I 2022-11-10 18:38:17] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1330/1522
[I 2022-11-10 18:38:17] ...trap/engines/engine.py.325:       elapsed: 0:36:06 | left: 0:04:38
[I 2022-11-10 18:38:17] ...trap/engines/engine.py.326:       process: 1.07845 | load: 0.00017
[I 2022-11-10 18:38:31] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1340/1522
[I 2022-11-10 18:38:31] ...trap/engines/engine.py.325:       elapsed: 0:36:21 | left: 0:04:18
[I 2022-11-10 18:38:31] ...trap/engines/engine.py.326:       process: 1.07716 | load: 0.00015
[I 2022-11-10 18:38:47] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1350/1522
[I 2022-11-10 18:38:47] ...trap/engines/engine.py.325:       elapsed: 0:36:36 | left: 0:04:05
[I 2022-11-10 18:38:47] ...trap/engines/engine.py.326:       process: 1.39836 | load: 0.33022
[I 2022-11-10 18:39:04] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1360/1522
[I 2022-11-10 18:39:04] ...trap/engines/engine.py.325:       elapsed: 0:36:54 | left: 0:04:34
[I 2022-11-10 18:39:04] ...trap/engines/engine.py.326:       process: 1.06216 | load: 0.00008
[I 2022-11-10 18:39:20] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1370/1522
[I 2022-11-10 18:39:20] ...trap/engines/engine.py.325:       elapsed: 0:37:09 | left: 0:03:48
[I 2022-11-10 18:39:20] ...trap/engines/engine.py.326:       process: 1.07196 | load: 0.00013
[I 2022-11-10 18:39:36] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1380/1522
[I 2022-11-10 18:39:36] ...trap/engines/engine.py.325:       elapsed: 0:37:26 | left: 0:03:35
[I 2022-11-10 18:39:37] ...trap/engines/engine.py.326:       process: 1.06815 | load: 0.00013
[I 2022-11-10 18:39:50] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1390/1522
[I 2022-11-10 18:39:50] ...trap/engines/engine.py.325:       elapsed: 0:37:39 | left: 0:02:56
[I 2022-11-10 18:39:50] ...trap/engines/engine.py.326:       process: 1.07937 | load: 0.00014
[I 2022-11-10 18:40:08] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1400/1522
[I 2022-11-10 18:40:08] ...trap/engines/engine.py.325:       elapsed: 0:37:57 | left: 0:03:41
[I 2022-11-10 18:40:08] ...trap/engines/engine.py.326:       process: 1.07212 | load: 0.00007
[I 2022-11-10 18:40:24] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1410/1522
[I 2022-11-10 18:40:24] ...trap/engines/engine.py.325:       elapsed: 0:38:13 | left: 0:02:47
[I 2022-11-10 18:40:24] ...trap/engines/engine.py.326:       process: 1.11792 | load: 0.05098
[I 2022-11-10 18:40:41] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1420/1522
[I 2022-11-10 18:40:41] ...trap/engines/engine.py.325:       elapsed: 0:38:30 | left: 0:02:58
[I 2022-11-10 18:40:41] ...trap/engines/engine.py.326:       process: 1.07323 | load: 0.00007
[I 2022-11-10 18:40:55] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1430/1522
[I 2022-11-10 18:40:55] ...trap/engines/engine.py.325:       elapsed: 0:38:44 | left: 0:02:08
[I 2022-11-10 18:40:55] ...trap/engines/engine.py.326:       process: 1.06948 | load: 0.00016
[I 2022-11-10 18:41:12] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1440/1522
[I 2022-11-10 18:41:12] ...trap/engines/engine.py.325:       elapsed: 0:39:01 | left: 0:02:18
[I 2022-11-10 18:41:12] ...trap/engines/engine.py.326:       process: 1.07508 | load: 0.00015
[I 2022-11-10 18:41:27] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1450/1522
[I 2022-11-10 18:41:27] ...trap/engines/engine.py.325:       elapsed: 0:39:16 | left: 0:01:45
[I 2022-11-10 18:41:27] ...trap/engines/engine.py.326:       process: 1.07940 | load: 0.00016
[I 2022-11-10 18:41:44] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1460/1522
[I 2022-11-10 18:41:44] ...trap/engines/engine.py.325:       elapsed: 0:39:33 | left: 0:01:42
[I 2022-11-10 18:41:44] ...trap/engines/engine.py.326:       process: 1.67780 | load: 0.60909
[I 2022-11-10 18:41:59] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1470/1522
[I 2022-11-10 18:41:59] ...trap/engines/engine.py.325:       elapsed: 0:39:48 | left: 0:01:15
[I 2022-11-10 18:41:59] ...trap/engines/engine.py.326:       process: 1.06973 | load: 0.00013
[I 2022-11-10 18:42:16] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1480/1522
[I 2022-11-10 18:42:16] ...trap/engines/engine.py.325:       elapsed: 0:40:05 | left: 0:01:11
[I 2022-11-10 18:42:16] ...trap/engines/engine.py.326:       process: 1.07141 | load: 0.00007
[I 2022-11-10 18:42:30] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1490/1522
[I 2022-11-10 18:42:30] ...trap/engines/engine.py.325:       elapsed: 0:40:19 | left: 0:00:44
[I 2022-11-10 18:42:30] ...trap/engines/engine.py.326:       process: 1.07645 | load: 0.00013
[I 2022-11-10 18:42:47] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1500/1522
[I 2022-11-10 18:42:47] ...trap/engines/engine.py.325:       elapsed: 0:40:36 | left: 0:00:35
[I 2022-11-10 18:42:47] ...trap/engines/engine.py.326:       process: 1.36747 | load: 0.29308
[I 2022-11-10 18:43:02] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1510/1522
[I 2022-11-10 18:43:02] ...trap/engines/engine.py.325:       elapsed: 0:40:51 | left: 0:00:16
[I 2022-11-10 18:43:02] ...trap/engines/engine.py.326:       process: 1.06724 | load: 0.00013
[I 2022-11-10 18:43:19] ...trap/engines/engine.py.321: eval: epoch 59 | batch 1520/1522
[I 2022-11-10 18:43:19] ...trap/engines/engine.py.325:       elapsed: 0:41:08 | left: 0:00:03
[I 2022-11-10 18:43:19] ...trap/engines/engine.py.326:       process: 1.08170 | load: 0.00007
[S 2022-11-10 18:43:21] ...trap/engines/engine.py.113: eval_epoch.epoch: 59
[I 2022-11-10 18:43:21] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 5 x 10000 matchs
[S 2022-11-10 18:46:53] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 8.00
[S 2022-11-10 18:46:53] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 9.20
[S 2022-11-10 18:46:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.197580
[S 2022-11-10 18:46:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.434180
[S 2022-11-10 18:46:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.550200
[S 2022-11-10 18:46:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.179800
[S 2022-11-10 18:46:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.406700
[S 2022-11-10 18:46:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.522260
[S 2022-11-10 18:46:53] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000
[S 2022-11-10 18:46:53] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.400000
[S 2022-11-10 18:46:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.002721
[S 2022-11-10 18:46:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.004332
[S 2022-11-10 18:46:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.004227
[S 2022-11-10 18:46:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.003619
[S 2022-11-10 18:46:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.004190
[S 2022-11-10 18:46:53] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.004103
[I 2022-11-10 18:48:05] ...trap/engines/engine.py.115: Ending evaluation procedures
[W 2022-11-10 18:48:10] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json'
[W 2022-11-10 18:48:10] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[W 2022-11-10 18:48:10] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[I 2022-11-10 18:48:21] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_recipe1m_13m_orig_ids/view.html