[I 2022-10-28 12:33:02] ...ytorch/bootstrap/run.py.73: Saving environment info
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75: options: 
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:   dataset:
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     aux_kw_path:
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       test: /home/mshukor/data/recipe1m/clip_da/layer1_test_titles_kw.json
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       train: /home/mshukor/data/recipe1m/clip_da/layer1_train_titles_kw.json
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       val: /home/mshukor/data/recipe1m/clip_da/layer1_val_titles_kw.json
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     batch_sampler: triplet_classif
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     batch_size: 100
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     database: lmdb
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     dir: /home/mshukor/data/recipe1m
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     eval_split: test
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     freq_mismatch: 0.0
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     image_from: database
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     import: recipe1m.datasets.factory
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     interchange_ingrd_instr: None
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     kw_path:
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       test: /home/mshukor/data/recipe1m/clip_da/layer1_test_ingr_kw.json
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       train: /home/mshukor/data/recipe1m/clip_da/layer1_train_ingr_kw.json
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       val: /home/mshukor/data/recipe1m/clip_da/layer1_val_ingr_kw.json
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     name: recipe1m
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     nb_classes: 1048
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     nb_threads: 4
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     randkw_p: 0.3
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     randkw_p_aux: 0.5
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     remove_list: None
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     tokenized_raw_text: True
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     train_split: None
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     use_vcs: True
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:   engine:
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     name: logger
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     nb_epochs: 120
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     print_freq: 10
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     saving_criteria: ['eval_epoch.metric.med_im2recipe_mean:min', 'eval_epoch.metric.recall_at_1_im2recipe_mean:max']
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:   exp:
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     checkpoint: None
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     dir: /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     resume: best_eval_epoch.metric.recall_at_1_im2recipe_mean
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:   misc:
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     cuda: True
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     data_parrallel: True
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     logs_name: None
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     overrite: False
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     seed: 1338
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:   model:
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     criterion:
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:         tri_cross_encoder: False
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       itm_loss_weight: 1
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       keep_background: False
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       retrieval_strategy:
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:         aggregation: valid
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:         get_tokens: True
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:         margin: 0.05
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:         margin_params:
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:           increment: 0.005
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:           increment_margin: True
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:           max_margin: 0.3
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:         name: triplet
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:         nb_samples: 9999
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:         sampling: max_negative
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:         substrategy: ['IRR', 'RII', 'SIRR', 'SRII']
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:         substrategy_weights: [1.0, 1.0, 0.1, 0.1]
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     import: recipe1m.models.factory
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     metric:
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       k_test: 10
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       nb_bags: 10
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       nb_matchs_per_bag: 1000
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       retrieval_dir: /tmp/recipe1m
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       save_ids: False
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       trijoint: True
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     name: trijoint
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     network:
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       activations: ['tanh', 'normalize']
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       aux_kwords: True
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       aux_kwords_encoder: True
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       avg_concat: True
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       bert_config: /home/mshukor/tfood/recipe1m/models/networks/recipe_networks/config_bert_albef.json
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       cat_pos: 0
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       checkpoint: None
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       class_attention: False
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       class_attention_fus: False
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       cls_token: False
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       cls_token_fus: False
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       cross_decoder_image: True
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       cross_decoder_recipe: True
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:         class_attention: False
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:         cls_token: False
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:         cross_attention: False
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:         cross_decoder: True
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:         double_decoder: False
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:         get_tokens: False
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:         load_backbones: False
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:         n_heads: 4
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:         n_layers: 4
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:         query_img: False
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       cross_transformer: False
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       dim_emb: 1024
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       dim_image_out: 2048
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       dim_ingr_out: 300
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       dim_instr_in: 1024
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       dim_instr_out: 1024
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       freeze_im: False
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       freeze_rec: False
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       get_tokens: True
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       get_tokens_cross_decoder_recipe: True
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       hidden_size: 512
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       image_backbone_name: clip
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       image_res: 224
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       kwords_same_level: False
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       last_cat_aux: True
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       n_heads: 4
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       n_heads_cross: 4
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       n_heads_cross_image: 4
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       n_heads_single: 4
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       n_layers: 2
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       n_layers_cross: 2
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       n_layers_cross_image: 1
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       n_layers_single: 2
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       num_hidden_layers_kw: 2
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       old: True
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       path_ingrs: /home/mshukor/data/recipe1m/text/vocab.pkl
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       path_vocab: /home/mshukor/data/recipe1m/text/vocab_all.txt
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       recipe_encoder: tfoodvicha-transformer
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       remove_additional_embedding: ['proj_image']
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       text_encoder: bert-base-uncased
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       vision_width: 768
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       vit_all_tokens: True
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       with_ingrs: True
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       with_instrs: True
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:       with_titles: True
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     with_classif: False
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:   optimizer:
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     batch_size_factor: None
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     clip_grad: 8.0
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     import: recipe1m.optimizers.factory
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     lr: 1e-05
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     lr_img: 1e-06
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     name: trijoint_fixed_fine_tune
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:     switch_epoch: 20
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:   path_opts: /home/mshukor/tfood/recipe1m/options/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_noimgproj.yaml
[S 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.75:   view: ['logs:train_epoch.loss', 'logs:train_epoch.bad_pairs', 'logs:eval_epoch.metric.med_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_1_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_5_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_10_im2recipe_mean', 'logs:eval_epoch.metric.med_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_1_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_5_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_10_recipe2im_mean', 'logs:optimizer.is_optimizer_recipe&image', 'logs:optimizer.total_norm']
[I 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.76: posix.uname_result(sysname='Linux', nodename='DGX', release='5.4.0-124-generic', version='#140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022', machine='x86_64')
[I 2022-10-28 12:33:03] ...ytorch/bootstrap/run.py.80: Available GPUs: [0]
[I 2022-10-28 12:33:03] ...strap/engines/factory.py.9: Creating engine...
[I 2022-10-28 12:33:03] ...trap/datasets/factory.py.8: Creating dataset...
[I 2022-10-28 12:33:05] ...m/datasets/recipe1m.py.152: Load VCs...
[I 2022-10-28 12:33:05] ...m/datasets/recipe1m.py.165: randkw_p... None
[I 2022-10-28 12:34:08] ...m/datasets/recipe1m.py.310: recipe elements to remove: None
[I 2022-10-28 12:34:08] ...rap/datasets/factory.py.21: Evaluation will take place on testset (51303 items)
[I 2022-10-28 12:34:08] ...strap/models/factory.py.21: Creating model...
[I 2022-10-28 12:34:09] ...e_networks/networks.py.203: Loading CLIP Visual model...
[I 2022-10-28 12:34:12] ...e_networks/networks.py.145: cuda
[I 2022-10-28 12:34:13] ...networks/tfood_vicha.py.49: Load VCs encoder...
[I 2022-10-28 12:34:16] ...els/metrics/trijoint.py.46: self.save_ids False
[I 2022-10-28 12:34:16] ...strap/models/factory.py.44: Enabling CUDA mode...
[I 2022-10-28 12:34:18] ...etworks/tfood_vicha.py.184: training aux kw modules
[I 2022-10-28 12:34:18] ...etworks/tfood_vicha.py.184: training aux kw modules
[I 2022-10-28 12:34:18] ...otstrap/views/factory.py.8: Creating views...
[I 2022-10-28 12:34:18] ...strap/engines/engine.py.98: Loading best_eval_epoch.metric.recall_at_1_im2recipe_mean checkpoint
[I 2022-10-28 12:34:18] ...trap/engines/engine.py.440: Loading model...
[I 2022-10-28 12:34:19] ...trap/engines/engine.py.453: Loading engine...
[I 2022-10-28 12:34:19] ...trap/engines/engine.py.108: Launching evaluation procedures
[I 2022-10-28 12:34:19] ...trap/engines/engine.py.268: Evaluating model on testset for epoch 116
[I 2022-10-28 12:34:19] ...m/datasets/recipe1m.py.400: Dataset will be sampled with "random" batch_sampler.
[I 2022-10-28 12:34:22] ...trap/engines/engine.py.321: eval: epoch 116 | batch 0/512
[I 2022-10-28 12:34:22] ...trap/engines/engine.py.325:       elapsed: 0:00:02 | left: 0:23:25
[I 2022-10-28 12:34:22] ...trap/engines/engine.py.326:       process: 2.74498 | load: 1.49981
[I 2022-10-28 12:34:25] ...trap/engines/engine.py.321: eval: epoch 116 | batch 10/512
[I 2022-10-28 12:34:25] ...trap/engines/engine.py.325:       elapsed: 0:00:06 | left: 0:04:53
[I 2022-10-28 12:34:25] ...trap/engines/engine.py.326:       process: 0.32213 | load: 0.00016
[I 2022-10-28 12:34:29] ...trap/engines/engine.py.321: eval: epoch 116 | batch 20/512
[I 2022-10-28 12:34:29] ...trap/engines/engine.py.325:       elapsed: 0:00:09 | left: 0:02:54
[I 2022-10-28 12:34:29] ...trap/engines/engine.py.326:       process: 0.32110 | load: 0.00014
[I 2022-10-28 12:34:32] ...trap/engines/engine.py.321: eval: epoch 116 | batch 30/512
[I 2022-10-28 12:34:32] ...trap/engines/engine.py.325:       elapsed: 0:00:12 | left: 0:02:39
[I 2022-10-28 12:34:32] ...trap/engines/engine.py.326:       process: 0.32329 | load: 0.00015
[I 2022-10-28 12:34:35] ...trap/engines/engine.py.321: eval: epoch 116 | batch 40/512
[I 2022-10-28 12:34:35] ...trap/engines/engine.py.325:       elapsed: 0:00:15 | left: 0:02:34
[I 2022-10-28 12:34:35] ...trap/engines/engine.py.326:       process: 0.32546 | load: 0.00015
[I 2022-10-28 12:34:38] ...trap/engines/engine.py.321: eval: epoch 116 | batch 50/512
[I 2022-10-28 12:34:38] ...trap/engines/engine.py.325:       elapsed: 0:00:19 | left: 0:02:30
[I 2022-10-28 12:34:38] ...trap/engines/engine.py.326:       process: 0.32439 | load: 0.00019
[I 2022-10-28 12:34:42] ...trap/engines/engine.py.321: eval: epoch 116 | batch 60/512
[I 2022-10-28 12:34:42] ...trap/engines/engine.py.325:       elapsed: 0:00:22 | left: 0:02:32
[I 2022-10-28 12:34:42] ...trap/engines/engine.py.326:       process: 0.38639 | load: 0.00015
[I 2022-10-28 12:34:45] ...trap/engines/engine.py.321: eval: epoch 116 | batch 70/512
[I 2022-10-28 12:34:45] ...trap/engines/engine.py.325:       elapsed: 0:00:25 | left: 0:02:26
[I 2022-10-28 12:34:45] ...trap/engines/engine.py.326:       process: 0.32558 | load: 0.00023
[I 2022-10-28 12:34:48] ...trap/engines/engine.py.321: eval: epoch 116 | batch 80/512
[I 2022-10-28 12:34:48] ...trap/engines/engine.py.325:       elapsed: 0:00:29 | left: 0:02:21
[I 2022-10-28 12:34:48] ...trap/engines/engine.py.326:       process: 0.32484 | load: 0.00016
[I 2022-10-28 12:34:52] ...trap/engines/engine.py.321: eval: epoch 116 | batch 90/512
[I 2022-10-28 12:34:52] ...trap/engines/engine.py.325:       elapsed: 0:00:32 | left: 0:02:21
[I 2022-10-28 12:34:52] ...trap/engines/engine.py.326:       process: 0.35344 | load: 0.00016
[I 2022-10-28 12:34:55] ...trap/engines/engine.py.321: eval: epoch 116 | batch 100/512
[I 2022-10-28 12:34:55] ...trap/engines/engine.py.325:       elapsed: 0:00:35 | left: 0:02:12
[I 2022-10-28 12:34:55] ...trap/engines/engine.py.326:       process: 0.28786 | load: 0.00016
[I 2022-10-28 12:34:58] ...trap/engines/engine.py.321: eval: epoch 116 | batch 110/512
[I 2022-10-28 12:34:58] ...trap/engines/engine.py.325:       elapsed: 0:00:38 | left: 0:01:57
[I 2022-10-28 12:34:58] ...trap/engines/engine.py.326:       process: 0.28697 | load: 0.00014
[I 2022-10-28 12:35:01] ...trap/engines/engine.py.321: eval: epoch 116 | batch 120/512
[I 2022-10-28 12:35:01] ...trap/engines/engine.py.325:       elapsed: 0:00:41 | left: 0:01:53
[I 2022-10-28 12:35:01] ...trap/engines/engine.py.326:       process: 0.28967 | load: 0.00014
[I 2022-10-28 12:35:04] ...trap/engines/engine.py.321: eval: epoch 116 | batch 130/512
[I 2022-10-28 12:35:04] ...trap/engines/engine.py.325:       elapsed: 0:00:44 | left: 0:01:51
[I 2022-10-28 12:35:04] ...trap/engines/engine.py.326:       process: 0.29038 | load: 0.00017
[I 2022-10-28 12:35:07] ...trap/engines/engine.py.321: eval: epoch 116 | batch 140/512
[I 2022-10-28 12:35:07] ...trap/engines/engine.py.325:       elapsed: 0:00:47 | left: 0:01:48
[I 2022-10-28 12:35:07] ...trap/engines/engine.py.326:       process: 0.30486 | load: 0.00015
[I 2022-10-28 12:35:09] ...trap/engines/engine.py.321: eval: epoch 116 | batch 150/512
[I 2022-10-28 12:35:09] ...trap/engines/engine.py.325:       elapsed: 0:00:50 | left: 0:01:44
[I 2022-10-28 12:35:09] ...trap/engines/engine.py.326:       process: 0.28369 | load: 0.00016
[I 2022-10-28 12:35:12] ...trap/engines/engine.py.321: eval: epoch 116 | batch 160/512
[I 2022-10-28 12:35:12] ...trap/engines/engine.py.325:       elapsed: 0:00:52 | left: 0:01:41
[I 2022-10-28 12:35:12] ...trap/engines/engine.py.326:       process: 0.29200 | load: 0.00013
[I 2022-10-28 12:35:15] ...trap/engines/engine.py.321: eval: epoch 116 | batch 170/512
[I 2022-10-28 12:35:15] ...trap/engines/engine.py.325:       elapsed: 0:00:55 | left: 0:01:37
[I 2022-10-28 12:35:15] ...trap/engines/engine.py.326:       process: 0.29057 | load: 0.00017
[I 2022-10-28 12:35:18] ...trap/engines/engine.py.321: eval: epoch 116 | batch 180/512
[I 2022-10-28 12:35:18] ...trap/engines/engine.py.325:       elapsed: 0:00:58 | left: 0:01:34
[I 2022-10-28 12:35:18] ...trap/engines/engine.py.326:       process: 0.28384 | load: 0.00013
[I 2022-10-28 12:35:21] ...trap/engines/engine.py.321: eval: epoch 116 | batch 190/512
[I 2022-10-28 12:35:21] ...trap/engines/engine.py.325:       elapsed: 0:01:01 | left: 0:01:32
[I 2022-10-28 12:35:21] ...trap/engines/engine.py.326:       process: 0.28558 | load: 0.00017
[I 2022-10-28 12:35:24] ...trap/engines/engine.py.321: eval: epoch 116 | batch 200/512
[I 2022-10-28 12:35:24] ...trap/engines/engine.py.325:       elapsed: 0:01:04 | left: 0:01:29
[I 2022-10-28 12:35:24] ...trap/engines/engine.py.326:       process: 0.29397 | load: 0.00013
[I 2022-10-28 12:35:27] ...trap/engines/engine.py.321: eval: epoch 116 | batch 210/512
[I 2022-10-28 12:35:27] ...trap/engines/engine.py.325:       elapsed: 0:01:07 | left: 0:01:27
[I 2022-10-28 12:35:27] ...trap/engines/engine.py.326:       process: 0.29133 | load: 0.00014
[I 2022-10-28 12:35:30] ...trap/engines/engine.py.321: eval: epoch 116 | batch 220/512
[I 2022-10-28 12:35:30] ...trap/engines/engine.py.325:       elapsed: 0:01:10 | left: 0:01:24
[I 2022-10-28 12:35:30] ...trap/engines/engine.py.326:       process: 0.28953 | load: 0.00014
[I 2022-10-28 12:35:33] ...trap/engines/engine.py.321: eval: epoch 116 | batch 230/512
[I 2022-10-28 12:35:33] ...trap/engines/engine.py.325:       elapsed: 0:01:13 | left: 0:01:21
[I 2022-10-28 12:35:33] ...trap/engines/engine.py.326:       process: 0.28458 | load: 0.00017
[I 2022-10-28 12:35:35] ...trap/engines/engine.py.321: eval: epoch 116 | batch 240/512
[I 2022-10-28 12:35:35] ...trap/engines/engine.py.325:       elapsed: 0:01:16 | left: 0:01:18
[I 2022-10-28 12:35:35] ...trap/engines/engine.py.326:       process: 0.28932 | load: 0.00012
[I 2022-10-28 12:35:38] ...trap/engines/engine.py.321: eval: epoch 116 | batch 250/512
[I 2022-10-28 12:35:38] ...trap/engines/engine.py.325:       elapsed: 0:01:18 | left: 0:01:15
[I 2022-10-28 12:35:38] ...trap/engines/engine.py.326:       process: 0.29089 | load: 0.00017
[I 2022-10-28 12:35:41] ...trap/engines/engine.py.321: eval: epoch 116 | batch 260/512
[I 2022-10-28 12:35:41] ...trap/engines/engine.py.325:       elapsed: 0:01:21 | left: 0:01:12
[I 2022-10-28 12:35:41] ...trap/engines/engine.py.326:       process: 0.28758 | load: 0.00016
[I 2022-10-28 12:35:44] ...trap/engines/engine.py.321: eval: epoch 116 | batch 270/512
[I 2022-10-28 12:35:44] ...trap/engines/engine.py.325:       elapsed: 0:01:24 | left: 0:01:09
[I 2022-10-28 12:35:44] ...trap/engines/engine.py.326:       process: 0.28697 | load: 0.00016
[I 2022-10-28 12:35:47] ...trap/engines/engine.py.321: eval: epoch 116 | batch 280/512
[I 2022-10-28 12:35:47] ...trap/engines/engine.py.325:       elapsed: 0:01:27 | left: 0:01:06
[I 2022-10-28 12:35:47] ...trap/engines/engine.py.326:       process: 0.28906 | load: 0.00018
[I 2022-10-28 12:35:50] ...trap/engines/engine.py.321: eval: epoch 116 | batch 290/512
[I 2022-10-28 12:35:50] ...trap/engines/engine.py.325:       elapsed: 0:01:30 | left: 0:01:03
[I 2022-10-28 12:35:50] ...trap/engines/engine.py.326:       process: 0.28494 | load: 0.00015
[I 2022-10-28 12:35:53] ...trap/engines/engine.py.321: eval: epoch 116 | batch 300/512
[I 2022-10-28 12:35:53] ...trap/engines/engine.py.325:       elapsed: 0:01:33 | left: 0:01:01
[I 2022-10-28 12:35:53] ...trap/engines/engine.py.326:       process: 0.28712 | load: 0.00014
[I 2022-10-28 12:35:56] ...trap/engines/engine.py.321: eval: epoch 116 | batch 310/512
[I 2022-10-28 12:35:56] ...trap/engines/engine.py.325:       elapsed: 0:01:36 | left: 0:00:58
[I 2022-10-28 12:35:56] ...trap/engines/engine.py.326:       process: 0.28650 | load: 0.00015
[I 2022-10-28 12:35:58] ...trap/engines/engine.py.321: eval: epoch 116 | batch 320/512
[I 2022-10-28 12:35:58] ...trap/engines/engine.py.325:       elapsed: 0:01:39 | left: 0:00:55
[I 2022-10-28 12:35:58] ...trap/engines/engine.py.326:       process: 0.28491 | load: 0.00019
[I 2022-10-28 12:36:01] ...trap/engines/engine.py.321: eval: epoch 116 | batch 330/512
[I 2022-10-28 12:36:01] ...trap/engines/engine.py.325:       elapsed: 0:01:41 | left: 0:00:52
[I 2022-10-28 12:36:01] ...trap/engines/engine.py.326:       process: 0.28740 | load: 0.00014
[I 2022-10-28 12:36:04] ...trap/engines/engine.py.321: eval: epoch 116 | batch 340/512
[I 2022-10-28 12:36:04] ...trap/engines/engine.py.325:       elapsed: 0:01:44 | left: 0:00:49
[I 2022-10-28 12:36:04] ...trap/engines/engine.py.326:       process: 0.29285 | load: 0.00015
[I 2022-10-28 12:36:07] ...trap/engines/engine.py.321: eval: epoch 116 | batch 350/512
[I 2022-10-28 12:36:07] ...trap/engines/engine.py.325:       elapsed: 0:01:47 | left: 0:00:46
[I 2022-10-28 12:36:07] ...trap/engines/engine.py.326:       process: 0.28160 | load: 0.00017
[I 2022-10-28 12:36:10] ...trap/engines/engine.py.321: eval: epoch 116 | batch 360/512
[I 2022-10-28 12:36:10] ...trap/engines/engine.py.325:       elapsed: 0:01:50 | left: 0:00:43
[I 2022-10-28 12:36:10] ...trap/engines/engine.py.326:       process: 0.28817 | load: 0.00014
[I 2022-10-28 12:36:13] ...trap/engines/engine.py.321: eval: epoch 116 | batch 370/512
[I 2022-10-28 12:36:13] ...trap/engines/engine.py.325:       elapsed: 0:01:53 | left: 0:00:41
[I 2022-10-28 12:36:13] ...trap/engines/engine.py.326:       process: 0.30528 | load: 0.00015
[I 2022-10-28 12:36:16] ...trap/engines/engine.py.321: eval: epoch 116 | batch 380/512
[I 2022-10-28 12:36:16] ...trap/engines/engine.py.325:       elapsed: 0:01:56 | left: 0:00:38
[I 2022-10-28 12:36:16] ...trap/engines/engine.py.326:       process: 0.29138 | load: 0.00019
[I 2022-10-28 12:36:19] ...trap/engines/engine.py.321: eval: epoch 116 | batch 390/512
[I 2022-10-28 12:36:19] ...trap/engines/engine.py.325:       elapsed: 0:01:59 | left: 0:00:35
[I 2022-10-28 12:36:19] ...trap/engines/engine.py.326:       process: 0.28803 | load: 0.00015
[I 2022-10-28 12:36:22] ...trap/engines/engine.py.321: eval: epoch 116 | batch 400/512
[I 2022-10-28 12:36:22] ...trap/engines/engine.py.325:       elapsed: 0:02:02 | left: 0:00:32
[I 2022-10-28 12:36:22] ...trap/engines/engine.py.326:       process: 0.28716 | load: 0.00014
[I 2022-10-28 12:36:24] ...trap/engines/engine.py.321: eval: epoch 116 | batch 410/512
[I 2022-10-28 12:36:24] ...trap/engines/engine.py.325:       elapsed: 0:02:04 | left: 0:00:29
[I 2022-10-28 12:36:24] ...trap/engines/engine.py.326:       process: 0.29153 | load: 0.00018
[I 2022-10-28 12:36:27] ...trap/engines/engine.py.321: eval: epoch 116 | batch 420/512
[I 2022-10-28 12:36:27] ...trap/engines/engine.py.325:       elapsed: 0:02:07 | left: 0:00:26
[I 2022-10-28 12:36:27] ...trap/engines/engine.py.326:       process: 0.28942 | load: 0.00013
[I 2022-10-28 12:36:30] ...trap/engines/engine.py.321: eval: epoch 116 | batch 430/512
[I 2022-10-28 12:36:30] ...trap/engines/engine.py.325:       elapsed: 0:02:10 | left: 0:00:23
[I 2022-10-28 12:36:30] ...trap/engines/engine.py.326:       process: 0.28211 | load: 0.00017
[I 2022-10-28 12:36:33] ...trap/engines/engine.py.321: eval: epoch 116 | batch 440/512
[I 2022-10-28 12:36:33] ...trap/engines/engine.py.325:       elapsed: 0:02:13 | left: 0:00:20
[I 2022-10-28 12:36:33] ...trap/engines/engine.py.326:       process: 0.28998 | load: 0.00014
[I 2022-10-28 12:36:36] ...trap/engines/engine.py.321: eval: epoch 116 | batch 450/512
[I 2022-10-28 12:36:36] ...trap/engines/engine.py.325:       elapsed: 0:02:16 | left: 0:00:17
[I 2022-10-28 12:36:36] ...trap/engines/engine.py.326:       process: 0.28268 | load: 0.00018
[I 2022-10-28 12:36:39] ...trap/engines/engine.py.321: eval: epoch 116 | batch 460/512
[I 2022-10-28 12:36:39] ...trap/engines/engine.py.325:       elapsed: 0:02:19 | left: 0:00:14
[I 2022-10-28 12:36:39] ...trap/engines/engine.py.326:       process: 0.28378 | load: 0.00017
[I 2022-10-28 12:36:42] ...trap/engines/engine.py.321: eval: epoch 116 | batch 470/512
[I 2022-10-28 12:36:42] ...trap/engines/engine.py.325:       elapsed: 0:02:22 | left: 0:00:12
[I 2022-10-28 12:36:42] ...trap/engines/engine.py.326:       process: 0.28078 | load: 0.00018
[I 2022-10-28 12:36:45] ...trap/engines/engine.py.321: eval: epoch 116 | batch 480/512
[I 2022-10-28 12:36:45] ...trap/engines/engine.py.325:       elapsed: 0:02:25 | left: 0:00:09
[I 2022-10-28 12:36:45] ...trap/engines/engine.py.326:       process: 0.28970 | load: 0.00017
[I 2022-10-28 12:36:47] ...trap/engines/engine.py.321: eval: epoch 116 | batch 490/512
[I 2022-10-28 12:36:47] ...trap/engines/engine.py.325:       elapsed: 0:02:27 | left: 0:00:06
[I 2022-10-28 12:36:47] ...trap/engines/engine.py.326:       process: 0.28132 | load: 0.00018
[I 2022-10-28 12:36:50] ...trap/engines/engine.py.321: eval: epoch 116 | batch 500/512
[I 2022-10-28 12:36:50] ...trap/engines/engine.py.325:       elapsed: 0:02:30 | left: 0:00:03
[I 2022-10-28 12:36:50] ...trap/engines/engine.py.326:       process: 0.28938 | load: 0.00014
[I 2022-10-28 12:36:53] ...trap/engines/engine.py.321: eval: epoch 116 | batch 510/512
[I 2022-10-28 12:36:53] ...trap/engines/engine.py.325:       elapsed: 0:02:33 | left: 0:00:00
[I 2022-10-28 12:36:53] ...trap/engines/engine.py.326:       process: 0.28087 | load: 0.00012
[S 2022-10-28 12:36:54] ...trap/engines/engine.py.113: eval_epoch.epoch: 116
[I 2022-10-28 12:36:54] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs
[S 2022-10-28 12:36:58] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00
[S 2022-10-28 12:36:58] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00
[S 2022-10-28 12:36:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.746600
[S 2022-10-28 12:36:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.907100
[S 2022-10-28 12:36:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.931900
[S 2022-10-28 12:36:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.735600
[S 2022-10-28 12:36:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.904600
[S 2022-10-28 12:36:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.933000
[S 2022-10-28 12:36:58] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000
[S 2022-10-28 12:36:58] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000
[S 2022-10-28 12:36:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.011333
[S 2022-10-28 12:36:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009126
[S 2022-10-28 12:36:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007752
[S 2022-10-28 12:36:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.010604
[S 2022-10-28 12:36:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.006560
[S 2022-10-28 12:36:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006213
[I 2022-10-28 12:36:59] ...trap/engines/engine.py.115: Ending evaluation procedures
[W 2022-10-28 12:36:59] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json'
[W 2022-10-28 12:36:59] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[W 2022-10-28 12:36:59] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[I 2022-10-28 12:37:03] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html
[I 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.73: Saving environment info
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75: options: 
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:   dataset:
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     aux_kw_path:
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       test: /home/mshukor/data/recipe1m/clip_da/layer1_test_titles_kw.json
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       train: /home/mshukor/data/recipe1m/clip_da/layer1_train_titles_kw.json
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       val: /home/mshukor/data/recipe1m/clip_da/layer1_val_titles_kw.json
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     batch_sampler: triplet_classif
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     batch_size: 100
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     database: lmdb
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     dir: /home/mshukor/data/recipe1m
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     eval_split: test
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     freq_mismatch: 0.0
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     image_from: database
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     import: recipe1m.datasets.factory
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     interchange_ingrd_instr: None
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     kw_path:
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       test: /home/mshukor/data/recipe1m/clip_da/layer1_test_ingr_kw.json
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       train: /home/mshukor/data/recipe1m/clip_da/layer1_train_ingr_kw.json
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       val: /home/mshukor/data/recipe1m/clip_da/layer1_val_ingr_kw.json
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     name: recipe1m
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     nb_classes: 1048
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     nb_threads: 4
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     randkw_p: 0.3
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     randkw_p_aux: 0.5
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     remove_list: None
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     tokenized_raw_text: True
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     train_split: None
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     use_vcs: True
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:   engine:
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     name: logger
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     nb_epochs: 120
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     print_freq: 10
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     saving_criteria: ['eval_epoch.metric.med_im2recipe_mean:min', 'eval_epoch.metric.recall_at_1_im2recipe_mean:max']
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:   exp:
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     checkpoint: None
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     dir: /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     resume: best_eval_epoch.metric.recall_at_1_im2recipe_mean
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:   misc:
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     cuda: True
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     data_parrallel: True
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     logs_name: None
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     overrite: False
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     seed: 1338
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:   model:
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     criterion:
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:         tri_cross_encoder: False
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       itm_loss_weight: 1
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       keep_background: False
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       retrieval_strategy:
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:         aggregation: valid
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:         get_tokens: True
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:         margin: 0.05
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:         margin_params:
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:           increment: 0.005
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:           increment_margin: True
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:           max_margin: 0.3
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:         name: triplet
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:         nb_samples: 9999
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:         sampling: max_negative
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:         substrategy: ['IRR', 'RII', 'SIRR', 'SRII']
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:         substrategy_weights: [1.0, 1.0, 0.1, 0.1]
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     import: recipe1m.models.factory
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     metric:
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       k_test: 10
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       nb_bags: 5
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       nb_matchs_per_bag: 10000
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       retrieval_dir: /tmp/recipe1m
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       save_ids: False
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       trijoint: True
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     name: trijoint
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     network:
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       activations: ['tanh', 'normalize']
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       aux_kwords: True
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       aux_kwords_encoder: True
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       avg_concat: True
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       bert_config: /home/mshukor/tfood/recipe1m/models/networks/recipe_networks/config_bert_albef.json
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       cat_pos: 0
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       checkpoint: None
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       class_attention: False
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       class_attention_fus: False
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       cls_token: False
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       cls_token_fus: False
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       cross_decoder_image: True
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       cross_decoder_recipe: True
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:         class_attention: False
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:         cls_token: False
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:         cross_attention: False
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:         cross_decoder: True
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:         double_decoder: False
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:         get_tokens: False
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:         load_backbones: False
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:         n_heads: 4
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:         n_layers: 4
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:         query_img: False
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       cross_transformer: False
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       dim_emb: 1024
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       dim_image_out: 2048
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       dim_ingr_out: 300
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       dim_instr_in: 1024
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       dim_instr_out: 1024
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       freeze_im: False
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       freeze_rec: False
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       get_tokens: True
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       get_tokens_cross_decoder_recipe: True
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       hidden_size: 512
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       image_backbone_name: clip
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       image_res: 224
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       kwords_same_level: False
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       last_cat_aux: True
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       n_heads: 4
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       n_heads_cross: 4
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       n_heads_cross_image: 4
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       n_heads_single: 4
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       n_layers: 2
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       n_layers_cross: 2
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       n_layers_cross_image: 1
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       n_layers_single: 2
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       num_hidden_layers_kw: 2
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       old: True
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       path_ingrs: /home/mshukor/data/recipe1m/text/vocab.pkl
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       path_vocab: /home/mshukor/data/recipe1m/text/vocab_all.txt
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       recipe_encoder: tfoodvicha-transformer
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       remove_additional_embedding: ['proj_image']
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       text_encoder: bert-base-uncased
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       vision_width: 768
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       vit_all_tokens: True
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       with_ingrs: True
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       with_instrs: True
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:       with_titles: True
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     with_classif: False
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:   optimizer:
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     batch_size_factor: None
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     clip_grad: 8.0
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     import: recipe1m.optimizers.factory
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     lr: 1e-05
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     lr_img: 1e-06
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     name: trijoint_fixed_fine_tune
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:     switch_epoch: 20
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:   path_opts: /home/mshukor/tfood/recipe1m/options/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_noimgproj.yaml
[S 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.75:   view: ['logs:train_epoch.loss', 'logs:train_epoch.bad_pairs', 'logs:eval_epoch.metric.med_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_1_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_5_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_10_im2recipe_mean', 'logs:eval_epoch.metric.med_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_1_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_5_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_10_recipe2im_mean', 'logs:optimizer.is_optimizer_recipe&image', 'logs:optimizer.total_norm']
[I 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.76: posix.uname_result(sysname='Linux', nodename='DGX', release='5.4.0-124-generic', version='#140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022', machine='x86_64')
[I 2022-10-28 12:42:21] ...ytorch/bootstrap/run.py.80: Available GPUs: [0]
[I 2022-10-28 12:42:21] ...strap/engines/factory.py.9: Creating engine...
[I 2022-10-28 12:42:21] ...trap/datasets/factory.py.8: Creating dataset...
[I 2022-10-28 12:42:23] ...m/datasets/recipe1m.py.152: Load VCs...
[I 2022-10-28 12:42:24] ...m/datasets/recipe1m.py.165: randkw_p... None
[I 2022-10-28 12:43:26] ...m/datasets/recipe1m.py.310: recipe elements to remove: None
[I 2022-10-28 12:43:26] ...rap/datasets/factory.py.21: Evaluation will take place on testset (51303 items)
[I 2022-10-28 12:43:26] ...strap/models/factory.py.21: Creating model...
[I 2022-10-28 12:43:27] ...e_networks/networks.py.203: Loading CLIP Visual model...
[I 2022-10-28 12:43:30] ...e_networks/networks.py.145: cuda
[I 2022-10-28 12:43:31] ...networks/tfood_vicha.py.49: Load VCs encoder...
[I 2022-10-28 12:43:34] ...els/metrics/trijoint.py.46: self.save_ids False
[I 2022-10-28 12:43:34] ...strap/models/factory.py.44: Enabling CUDA mode...
[I 2022-10-28 12:43:36] ...etworks/tfood_vicha.py.184: training aux kw modules
[I 2022-10-28 12:43:36] ...etworks/tfood_vicha.py.184: training aux kw modules
[I 2022-10-28 12:43:36] ...otstrap/views/factory.py.8: Creating views...
[I 2022-10-28 12:43:36] ...strap/engines/engine.py.98: Loading best_eval_epoch.metric.recall_at_1_im2recipe_mean checkpoint
[I 2022-10-28 12:43:36] ...trap/engines/engine.py.440: Loading model...
[I 2022-10-28 12:43:37] ...trap/engines/engine.py.453: Loading engine...
[I 2022-10-28 12:43:37] ...trap/engines/engine.py.108: Launching evaluation procedures
[I 2022-10-28 12:43:37] ...trap/engines/engine.py.268: Evaluating model on testset for epoch 116
[I 2022-10-28 12:43:37] ...m/datasets/recipe1m.py.400: Dataset will be sampled with "random" batch_sampler.
[I 2022-10-28 12:43:40] ...trap/engines/engine.py.321: eval: epoch 116 | batch 0/512
[I 2022-10-28 12:43:40] ...trap/engines/engine.py.325:       elapsed: 0:00:02 | left: 0:23:17
[I 2022-10-28 12:43:40] ...trap/engines/engine.py.326:       process: 2.72968 | load: 1.51144
[I 2022-10-28 12:43:43] ...trap/engines/engine.py.321: eval: epoch 116 | batch 10/512
[I 2022-10-28 12:43:43] ...trap/engines/engine.py.325:       elapsed: 0:00:06 | left: 0:04:53
[I 2022-10-28 12:43:43] ...trap/engines/engine.py.326:       process: 0.32429 | load: 0.00014
[I 2022-10-28 12:43:47] ...trap/engines/engine.py.321: eval: epoch 116 | batch 20/512
[I 2022-10-28 12:43:47] ...trap/engines/engine.py.325:       elapsed: 0:00:09 | left: 0:02:55
[I 2022-10-28 12:43:47] ...trap/engines/engine.py.326:       process: 0.33934 | load: 0.00015
[I 2022-10-28 12:43:50] ...trap/engines/engine.py.321: eval: epoch 116 | batch 30/512
[I 2022-10-28 12:43:50] ...trap/engines/engine.py.325:       elapsed: 0:00:12 | left: 0:02:37
[I 2022-10-28 12:43:50] ...trap/engines/engine.py.326:       process: 0.32172 | load: 0.00014
[I 2022-10-28 12:43:53] ...trap/engines/engine.py.321: eval: epoch 116 | batch 40/512
[I 2022-10-28 12:43:53] ...trap/engines/engine.py.325:       elapsed: 0:00:15 | left: 0:02:36
[I 2022-10-28 12:43:53] ...trap/engines/engine.py.326:       process: 0.32549 | load: 0.00017
[I 2022-10-28 12:43:56] ...trap/engines/engine.py.321: eval: epoch 116 | batch 50/512
[I 2022-10-28 12:43:56] ...trap/engines/engine.py.325:       elapsed: 0:00:19 | left: 0:02:30
[I 2022-10-28 12:43:56] ...trap/engines/engine.py.326:       process: 0.32388 | load: 0.00015
[I 2022-10-28 12:44:00] ...trap/engines/engine.py.321: eval: epoch 116 | batch 60/512
[I 2022-10-28 12:44:00] ...trap/engines/engine.py.325:       elapsed: 0:00:22 | left: 0:02:28
[I 2022-10-28 12:44:00] ...trap/engines/engine.py.326:       process: 0.33005 | load: 0.00017
[I 2022-10-28 12:44:03] ...trap/engines/engine.py.321: eval: epoch 116 | batch 70/512
[I 2022-10-28 12:44:03] ...trap/engines/engine.py.325:       elapsed: 0:00:25 | left: 0:02:26
[I 2022-10-28 12:44:03] ...trap/engines/engine.py.326:       process: 0.32966 | load: 0.00014
[I 2022-10-28 12:44:06] ...trap/engines/engine.py.321: eval: epoch 116 | batch 80/512
[I 2022-10-28 12:44:06] ...trap/engines/engine.py.325:       elapsed: 0:00:29 | left: 0:02:22
[I 2022-10-28 12:44:06] ...trap/engines/engine.py.326:       process: 0.32741 | load: 0.00016
[I 2022-10-28 12:44:10] ...trap/engines/engine.py.321: eval: epoch 116 | batch 90/512
[I 2022-10-28 12:44:10] ...trap/engines/engine.py.325:       elapsed: 0:00:32 | left: 0:02:19
[I 2022-10-28 12:44:10] ...trap/engines/engine.py.326:       process: 0.33236 | load: 0.00016
[I 2022-10-28 12:44:13] ...trap/engines/engine.py.321: eval: epoch 116 | batch 100/512
[I 2022-10-28 12:44:13] ...trap/engines/engine.py.325:       elapsed: 0:00:35 | left: 0:02:18
[I 2022-10-28 12:44:13] ...trap/engines/engine.py.326:       process: 0.34066 | load: 0.00018
[I 2022-10-28 12:44:16] ...trap/engines/engine.py.321: eval: epoch 116 | batch 110/512
[I 2022-10-28 12:44:16] ...trap/engines/engine.py.325:       elapsed: 0:00:39 | left: 0:02:14
[I 2022-10-28 12:44:16] ...trap/engines/engine.py.326:       process: 0.35225 | load: 0.00015
[I 2022-10-28 12:44:20] ...trap/engines/engine.py.321: eval: epoch 116 | batch 120/512
[I 2022-10-28 12:44:20] ...trap/engines/engine.py.325:       elapsed: 0:00:42 | left: 0:02:09
[I 2022-10-28 12:44:20] ...trap/engines/engine.py.326:       process: 0.32840 | load: 0.00018
[I 2022-10-28 12:44:23] ...trap/engines/engine.py.321: eval: epoch 116 | batch 130/512
[I 2022-10-28 12:44:23] ...trap/engines/engine.py.325:       elapsed: 0:00:45 | left: 0:02:07
[I 2022-10-28 12:44:23] ...trap/engines/engine.py.326:       process: 0.32330 | load: 0.00015
[I 2022-10-28 12:44:26] ...trap/engines/engine.py.321: eval: epoch 116 | batch 140/512
[I 2022-10-28 12:44:26] ...trap/engines/engine.py.325:       elapsed: 0:00:48 | left: 0:02:02
[I 2022-10-28 12:44:26] ...trap/engines/engine.py.326:       process: 0.33226 | load: 0.00016
[I 2022-10-28 12:44:30] ...trap/engines/engine.py.321: eval: epoch 116 | batch 150/512
[I 2022-10-28 12:44:30] ...trap/engines/engine.py.325:       elapsed: 0:00:52 | left: 0:02:00
[I 2022-10-28 12:44:30] ...trap/engines/engine.py.326:       process: 0.33391 | load: 0.00015
[I 2022-10-28 12:44:33] ...trap/engines/engine.py.321: eval: epoch 116 | batch 160/512
[I 2022-10-28 12:44:33] ...trap/engines/engine.py.325:       elapsed: 0:00:55 | left: 0:01:57
[I 2022-10-28 12:44:33] ...trap/engines/engine.py.326:       process: 0.32702 | load: 0.00016
[I 2022-10-28 12:44:36] ...trap/engines/engine.py.321: eval: epoch 116 | batch 170/512
[I 2022-10-28 12:44:36] ...trap/engines/engine.py.325:       elapsed: 0:00:58 | left: 0:01:52
[I 2022-10-28 12:44:36] ...trap/engines/engine.py.326:       process: 0.33774 | load: 0.00037
[I 2022-10-28 12:44:40] ...trap/engines/engine.py.321: eval: epoch 116 | batch 180/512
[I 2022-10-28 12:44:40] ...trap/engines/engine.py.325:       elapsed: 0:01:02 | left: 0:01:52
[I 2022-10-28 12:44:40] ...trap/engines/engine.py.326:       process: 0.36228 | load: 0.00020
[I 2022-10-28 12:44:43] ...trap/engines/engine.py.321: eval: epoch 116 | batch 190/512
[I 2022-10-28 12:44:43] ...trap/engines/engine.py.325:       elapsed: 0:01:05 | left: 0:01:49
[I 2022-10-28 12:44:43] ...trap/engines/engine.py.326:       process: 0.33341 | load: 0.00016
[I 2022-10-28 12:44:46] ...trap/engines/engine.py.321: eval: epoch 116 | batch 200/512
[I 2022-10-28 12:44:46] ...trap/engines/engine.py.325:       elapsed: 0:01:09 | left: 0:01:44
[I 2022-10-28 12:44:46] ...trap/engines/engine.py.326:       process: 0.33926 | load: 0.00018
[I 2022-10-28 12:44:50] ...trap/engines/engine.py.321: eval: epoch 116 | batch 210/512
[I 2022-10-28 12:44:50] ...trap/engines/engine.py.325:       elapsed: 0:01:12 | left: 0:01:40
[I 2022-10-28 12:44:50] ...trap/engines/engine.py.326:       process: 0.32986 | load: 0.00016
[I 2022-10-28 12:44:53] ...trap/engines/engine.py.321: eval: epoch 116 | batch 220/512
[I 2022-10-28 12:44:53] ...trap/engines/engine.py.325:       elapsed: 0:01:15 | left: 0:01:37
[I 2022-10-28 12:44:53] ...trap/engines/engine.py.326:       process: 0.33184 | load: 0.00017
[I 2022-10-28 12:44:56] ...trap/engines/engine.py.321: eval: epoch 116 | batch 230/512
[I 2022-10-28 12:44:56] ...trap/engines/engine.py.325:       elapsed: 0:01:18 | left: 0:01:33
[I 2022-10-28 12:44:56] ...trap/engines/engine.py.326:       process: 0.33369 | load: 0.00015
[I 2022-10-28 12:45:00] ...trap/engines/engine.py.321: eval: epoch 116 | batch 240/512
[I 2022-10-28 12:45:00] ...trap/engines/engine.py.325:       elapsed: 0:01:22 | left: 0:01:30
[I 2022-10-28 12:45:00] ...trap/engines/engine.py.326:       process: 0.33258 | load: 0.00016
[I 2022-10-28 12:45:03] ...trap/engines/engine.py.321: eval: epoch 116 | batch 250/512
[I 2022-10-28 12:45:03] ...trap/engines/engine.py.325:       elapsed: 0:01:25 | left: 0:01:27
[I 2022-10-28 12:45:03] ...trap/engines/engine.py.326:       process: 0.33376 | load: 0.00015
[I 2022-10-28 12:45:06] ...trap/engines/engine.py.321: eval: epoch 116 | batch 260/512
[I 2022-10-28 12:45:06] ...trap/engines/engine.py.325:       elapsed: 0:01:28 | left: 0:01:23
[I 2022-10-28 12:45:06] ...trap/engines/engine.py.326:       process: 0.33914 | load: 0.00017
[I 2022-10-28 12:45:10] ...trap/engines/engine.py.321: eval: epoch 116 | batch 270/512
[I 2022-10-28 12:45:10] ...trap/engines/engine.py.325:       elapsed: 0:01:32 | left: 0:01:20
[I 2022-10-28 12:45:10] ...trap/engines/engine.py.326:       process: 0.33530 | load: 0.00015
[I 2022-10-28 12:45:13] ...trap/engines/engine.py.321: eval: epoch 116 | batch 280/512
[I 2022-10-28 12:45:13] ...trap/engines/engine.py.325:       elapsed: 0:01:35 | left: 0:01:18
[I 2022-10-28 12:45:13] ...trap/engines/engine.py.326:       process: 0.33161 | load: 0.00017
[I 2022-10-28 12:45:16] ...trap/engines/engine.py.321: eval: epoch 116 | batch 290/512
[I 2022-10-28 12:45:16] ...trap/engines/engine.py.325:       elapsed: 0:01:38 | left: 0:01:13
[I 2022-10-28 12:45:16] ...trap/engines/engine.py.326:       process: 0.32230 | load: 0.00016
[I 2022-10-28 12:45:20] ...trap/engines/engine.py.321: eval: epoch 116 | batch 300/512
[I 2022-10-28 12:45:20] ...trap/engines/engine.py.325:       elapsed: 0:01:42 | left: 0:01:11
[I 2022-10-28 12:45:20] ...trap/engines/engine.py.326:       process: 0.33014 | load: 0.00020
[I 2022-10-28 12:45:23] ...trap/engines/engine.py.321: eval: epoch 116 | batch 310/512
[I 2022-10-28 12:45:23] ...trap/engines/engine.py.325:       elapsed: 0:01:45 | left: 0:01:08
[I 2022-10-28 12:45:23] ...trap/engines/engine.py.326:       process: 0.33685 | load: 0.00018
[I 2022-10-28 12:45:26] ...trap/engines/engine.py.321: eval: epoch 116 | batch 320/512
[I 2022-10-28 12:45:26] ...trap/engines/engine.py.325:       elapsed: 0:01:49 | left: 0:01:03
[I 2022-10-28 12:45:26] ...trap/engines/engine.py.326:       process: 0.32872 | load: 0.00018
[I 2022-10-28 12:45:30] ...trap/engines/engine.py.321: eval: epoch 116 | batch 330/512
[I 2022-10-28 12:45:30] ...trap/engines/engine.py.325:       elapsed: 0:01:52 | left: 0:01:00
[I 2022-10-28 12:45:30] ...trap/engines/engine.py.326:       process: 0.33037 | load: 0.00016
[I 2022-10-28 12:45:33] ...trap/engines/engine.py.321: eval: epoch 116 | batch 340/512
[I 2022-10-28 12:45:33] ...trap/engines/engine.py.325:       elapsed: 0:01:55 | left: 0:00:57
[I 2022-10-28 12:45:33] ...trap/engines/engine.py.326:       process: 0.33476 | load: 0.00018
[I 2022-10-28 12:45:36] ...trap/engines/engine.py.321: eval: epoch 116 | batch 350/512
[I 2022-10-28 12:45:36] ...trap/engines/engine.py.325:       elapsed: 0:01:59 | left: 0:00:53
[I 2022-10-28 12:45:36] ...trap/engines/engine.py.326:       process: 0.32969 | load: 0.00016
[I 2022-10-28 12:45:40] ...trap/engines/engine.py.321: eval: epoch 116 | batch 360/512
[I 2022-10-28 12:45:40] ...trap/engines/engine.py.325:       elapsed: 0:02:02 | left: 0:00:50
[I 2022-10-28 12:45:40] ...trap/engines/engine.py.326:       process: 0.32504 | load: 0.00017
[I 2022-10-28 12:45:43] ...trap/engines/engine.py.321: eval: epoch 116 | batch 370/512
[I 2022-10-28 12:45:43] ...trap/engines/engine.py.325:       elapsed: 0:02:05 | left: 0:00:47
[I 2022-10-28 12:45:43] ...trap/engines/engine.py.326:       process: 0.33097 | load: 0.00017
[I 2022-10-28 12:45:46] ...trap/engines/engine.py.321: eval: epoch 116 | batch 380/512
[I 2022-10-28 12:45:46] ...trap/engines/engine.py.325:       elapsed: 0:02:09 | left: 0:00:43
[I 2022-10-28 12:45:46] ...trap/engines/engine.py.326:       process: 0.33349 | load: 0.00016
[I 2022-10-28 12:45:50] ...trap/engines/engine.py.321: eval: epoch 116 | batch 390/512
[I 2022-10-28 12:45:50] ...trap/engines/engine.py.325:       elapsed: 0:02:12 | left: 0:00:40
[I 2022-10-28 12:45:50] ...trap/engines/engine.py.326:       process: 0.33016 | load: 0.00017
[I 2022-10-28 12:45:53] ...trap/engines/engine.py.321: eval: epoch 116 | batch 400/512
[I 2022-10-28 12:45:53] ...trap/engines/engine.py.325:       elapsed: 0:02:15 | left: 0:00:37
[I 2022-10-28 12:45:53] ...trap/engines/engine.py.326:       process: 0.33000 | load: 0.00017
[I 2022-10-28 12:45:56] ...trap/engines/engine.py.321: eval: epoch 116 | batch 410/512
[I 2022-10-28 12:45:56] ...trap/engines/engine.py.325:       elapsed: 0:02:19 | left: 0:00:33
[I 2022-10-28 12:45:56] ...trap/engines/engine.py.326:       process: 0.33483 | load: 0.00016
[I 2022-10-28 12:46:00] ...trap/engines/engine.py.321: eval: epoch 116 | batch 420/512
[I 2022-10-28 12:46:00] ...trap/engines/engine.py.325:       elapsed: 0:02:22 | left: 0:00:30
[I 2022-10-28 12:46:00] ...trap/engines/engine.py.326:       process: 0.33445 | load: 0.00017
[I 2022-10-28 12:46:03] ...trap/engines/engine.py.321: eval: epoch 116 | batch 430/512
[I 2022-10-28 12:46:03] ...trap/engines/engine.py.325:       elapsed: 0:02:25 | left: 0:00:27
[I 2022-10-28 12:46:03] ...trap/engines/engine.py.326:       process: 0.34647 | load: 0.00017
[I 2022-10-28 12:46:06] ...trap/engines/engine.py.321: eval: epoch 116 | batch 440/512
[I 2022-10-28 12:46:06] ...trap/engines/engine.py.325:       elapsed: 0:02:29 | left: 0:00:24
[I 2022-10-28 12:46:06] ...trap/engines/engine.py.326:       process: 0.33506 | load: 0.00020
[I 2022-10-28 12:46:10] ...trap/engines/engine.py.321: eval: epoch 116 | batch 450/512
[I 2022-10-28 12:46:10] ...trap/engines/engine.py.325:       elapsed: 0:02:32 | left: 0:00:20
[I 2022-10-28 12:46:10] ...trap/engines/engine.py.326:       process: 0.32312 | load: 0.00016
[I 2022-10-28 12:46:13] ...trap/engines/engine.py.321: eval: epoch 116 | batch 460/512
[I 2022-10-28 12:46:13] ...trap/engines/engine.py.325:       elapsed: 0:02:35 | left: 0:00:17
[I 2022-10-28 12:46:13] ...trap/engines/engine.py.326:       process: 0.33259 | load: 0.00017
[I 2022-10-28 12:46:16] ...trap/engines/engine.py.321: eval: epoch 116 | batch 470/512
[I 2022-10-28 12:46:16] ...trap/engines/engine.py.325:       elapsed: 0:02:39 | left: 0:00:13
[I 2022-10-28 12:46:16] ...trap/engines/engine.py.326:       process: 0.32888 | load: 0.00017
[I 2022-10-28 12:46:20] ...trap/engines/engine.py.321: eval: epoch 116 | batch 480/512
[I 2022-10-28 12:46:20] ...trap/engines/engine.py.325:       elapsed: 0:02:42 | left: 0:00:10
[I 2022-10-28 12:46:20] ...trap/engines/engine.py.326:       process: 0.33240 | load: 0.00016
[I 2022-10-28 12:46:23] ...trap/engines/engine.py.321: eval: epoch 116 | batch 490/512
[I 2022-10-28 12:46:23] ...trap/engines/engine.py.325:       elapsed: 0:02:45 | left: 0:00:07
[I 2022-10-28 12:46:23] ...trap/engines/engine.py.326:       process: 0.32795 | load: 0.00017
[I 2022-10-28 12:46:26] ...trap/engines/engine.py.321: eval: epoch 116 | batch 500/512
[I 2022-10-28 12:46:26] ...trap/engines/engine.py.325:       elapsed: 0:02:49 | left: 0:00:03
[I 2022-10-28 12:46:26] ...trap/engines/engine.py.326:       process: 0.28459 | load: 0.00019
[I 2022-10-28 12:46:29] ...trap/engines/engine.py.321: eval: epoch 116 | batch 510/512
[I 2022-10-28 12:46:29] ...trap/engines/engine.py.325:       elapsed: 0:02:51 | left: 0:00:00
[I 2022-10-28 12:46:29] ...trap/engines/engine.py.326:       process: 0.28080 | load: 0.00010
[S 2022-10-28 12:46:30] ...trap/engines/engine.py.113: eval_epoch.epoch: 116
[I 2022-10-28 12:46:30] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 5 x 10000 matchs
[S 2022-10-28 12:48:47] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 2.00
[S 2022-10-28 12:48:47] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 2.00
[S 2022-10-28 12:48:47] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.464460
[S 2022-10-28 12:48:47] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.730580
[S 2022-10-28 12:48:47] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.809300
[S 2022-10-28 12:48:47] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.452720
[S 2022-10-28 12:48:47] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.724480
[S 2022-10-28 12:48:47] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.807880
[S 2022-10-28 12:48:47] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000
[S 2022-10-28 12:48:47] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000
[S 2022-10-28 12:48:47] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.005709
[S 2022-10-28 12:48:47] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.003294
[S 2022-10-28 12:48:47] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.004174
[S 2022-10-28 12:48:47] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.004459
[S 2022-10-28 12:48:47] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.002967
[S 2022-10-28 12:48:47] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.002785
[I 2022-10-28 12:48:49] ...trap/engines/engine.py.115: Ending evaluation procedures
[W 2022-10-28 12:48:49] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json'
[W 2022-10-28 12:48:49] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[W 2022-10-28 12:48:49] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[I 2022-10-28 12:48:53] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html
[I 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.73: Saving environment info
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75: options: 
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:   dataset:
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     aux_kw_path:
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       test: /home/mshukor/data/recipe1m/clip_da/layer1_test_titles_kw.json
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       train: /home/mshukor/data/recipe1m/clip_da/layer1_train_titles_kw.json
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       val: /home/mshukor/data/recipe1m/clip_da/layer1_val_titles_kw.json
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     batch_sampler: triplet_classif
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     batch_size: 100
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     database: lmdb
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     dir: /home/mshukor/data/recipe1m
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     eval_split: test
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     freq_mismatch: 0.0
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     image_from: database
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     import: recipe1m.datasets.factory
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     interchange_ingrd_instr: None
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     kw_path:
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       test: /home/mshukor/data/recipe1m/clip_da/layer1_test_ingr_kw.json
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       train: /home/mshukor/data/recipe1m/clip_da/layer1_train_ingr_kw.json
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       val: /home/mshukor/data/recipe1m/clip_da/layer1_val_ingr_kw.json
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     name: recipe1m
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     nb_classes: 1048
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     nb_threads: 4
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     randkw_p: 0.3
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     randkw_p_aux: 0.5
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     remove_list: [instructions]
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     tokenized_raw_text: True
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     train_split: None
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     use_vcs: True
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:   engine:
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     name: logger
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     nb_epochs: 120
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     print_freq: 10
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     saving_criteria: ['eval_epoch.metric.med_im2recipe_mean:min', 'eval_epoch.metric.recall_at_1_im2recipe_mean:max']
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:   exp:
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     checkpoint: None
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     dir: /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     resume: best_eval_epoch.metric.recall_at_1_im2recipe_mean
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:   misc:
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     cuda: True
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     data_parrallel: True
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     logs_name: None
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     overrite: False
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     seed: 1338
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:   model:
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     criterion:
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:         tri_cross_encoder: False
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       itm_loss_weight: 1
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       keep_background: False
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       retrieval_strategy:
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:         aggregation: valid
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:         get_tokens: True
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:         margin: 0.05
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:         margin_params:
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:           increment: 0.005
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:           increment_margin: True
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:           max_margin: 0.3
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:         name: triplet
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:         nb_samples: 9999
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:         sampling: max_negative
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:         substrategy: ['IRR', 'RII', 'SIRR', 'SRII']
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:         substrategy_weights: [1.0, 1.0, 0.1, 0.1]
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     import: recipe1m.models.factory
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     metric:
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       k_test: 10
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       nb_bags: 10
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       nb_matchs_per_bag: 1000
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       retrieval_dir: /tmp/recipe1m
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       save_ids: False
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       trijoint: True
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     name: trijoint
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     network:
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       activations: ['tanh', 'normalize']
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       aux_kwords: True
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       aux_kwords_encoder: True
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       avg_concat: True
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       bert_config: /home/mshukor/tfood/recipe1m/models/networks/recipe_networks/config_bert_albef.json
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       cat_pos: 0
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       checkpoint: None
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       class_attention: False
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       class_attention_fus: False
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       cls_token: False
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       cls_token_fus: False
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       cross_decoder_image: True
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       cross_decoder_recipe: True
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:         class_attention: False
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:         cls_token: False
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:         cross_attention: False
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:         cross_decoder: True
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:         double_decoder: False
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:         get_tokens: False
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:         load_backbones: False
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:         n_heads: 4
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:         n_layers: 4
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:         query_img: False
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       cross_transformer: False
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       dim_emb: 1024
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       dim_image_out: 2048
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       dim_ingr_out: 300
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       dim_instr_in: 1024
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       dim_instr_out: 1024
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       freeze_im: False
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       freeze_rec: False
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       get_tokens: True
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       get_tokens_cross_decoder_recipe: True
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       hidden_size: 512
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       image_backbone_name: clip
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       image_res: 224
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       kwords_same_level: False
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       last_cat_aux: True
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       n_heads: 4
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       n_heads_cross: 4
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       n_heads_cross_image: 4
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       n_heads_single: 4
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       n_layers: 2
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       n_layers_cross: 2
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       n_layers_cross_image: 1
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       n_layers_single: 2
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       num_hidden_layers_kw: 2
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       old: True
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       path_ingrs: /home/mshukor/data/recipe1m/text/vocab.pkl
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       path_vocab: /home/mshukor/data/recipe1m/text/vocab_all.txt
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       recipe_encoder: tfoodvicha-transformer
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       remove_additional_embedding: ['proj_image']
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       text_encoder: bert-base-uncased
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       vision_width: 768
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       vit_all_tokens: True
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       with_ingrs: True
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       with_instrs: True
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:       with_titles: True
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     with_classif: False
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:   optimizer:
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     batch_size_factor: None
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     clip_grad: 8.0
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     import: recipe1m.optimizers.factory
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     lr: 1e-05
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     lr_img: 1e-06
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     name: trijoint_fixed_fine_tune
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:     switch_epoch: 20
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:   path_opts: /home/mshukor/tfood/recipe1m/options/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_noimgproj.yaml
[S 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.75:   view: ['logs:train_epoch.loss', 'logs:train_epoch.bad_pairs', 'logs:eval_epoch.metric.med_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_1_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_5_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_10_im2recipe_mean', 'logs:eval_epoch.metric.med_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_1_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_5_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_10_recipe2im_mean', 'logs:optimizer.is_optimizer_recipe&image', 'logs:optimizer.total_norm']
[I 2022-10-28 13:15:47] ...ytorch/bootstrap/run.py.76: posix.uname_result(sysname='Linux', nodename='DGX', release='5.4.0-124-generic', version='#140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022', machine='x86_64')
[I 2022-10-28 13:15:48] ...ytorch/bootstrap/run.py.80: Available GPUs: [0]
[I 2022-10-28 13:15:48] ...strap/engines/factory.py.9: Creating engine...
[I 2022-10-28 13:15:48] ...trap/datasets/factory.py.8: Creating dataset...
[I 2022-10-28 13:15:49] ...m/datasets/recipe1m.py.152: Load VCs...
[I 2022-10-28 13:15:50] ...m/datasets/recipe1m.py.165: randkw_p... None
[I 2022-10-28 13:16:53] ...m/datasets/recipe1m.py.310: recipe elements to remove: [instructions]
[I 2022-10-28 13:16:53] ...rap/datasets/factory.py.21: Evaluation will take place on testset (51303 items)
[I 2022-10-28 13:16:53] ...strap/models/factory.py.21: Creating model...
[I 2022-10-28 13:16:54] ...e_networks/networks.py.203: Loading CLIP Visual model...
[I 2022-10-28 13:16:57] ...e_networks/networks.py.145: cuda
[I 2022-10-28 13:16:58] ...networks/tfood_vicha.py.49: Load VCs encoder...
[I 2022-10-28 13:17:01] ...els/metrics/trijoint.py.46: self.save_ids False
[I 2022-10-28 13:17:01] ...strap/models/factory.py.44: Enabling CUDA mode...
[I 2022-10-28 13:17:03] ...etworks/tfood_vicha.py.184: training aux kw modules
[I 2022-10-28 13:17:03] ...etworks/tfood_vicha.py.184: training aux kw modules
[I 2022-10-28 13:17:03] ...otstrap/views/factory.py.8: Creating views...
[I 2022-10-28 13:17:04] ...strap/engines/engine.py.98: Loading best_eval_epoch.metric.recall_at_1_im2recipe_mean checkpoint
[I 2022-10-28 13:17:04] ...trap/engines/engine.py.440: Loading model...
[I 2022-10-28 13:17:05] ...trap/engines/engine.py.453: Loading engine...
[I 2022-10-28 13:17:05] ...trap/engines/engine.py.108: Launching evaluation procedures
[I 2022-10-28 13:17:05] ...trap/engines/engine.py.268: Evaluating model on testset for epoch 116
[I 2022-10-28 13:17:05] ...m/datasets/recipe1m.py.400: Dataset will be sampled with "random" batch_sampler.
[I 2022-10-28 13:17:07] ...trap/engines/engine.py.321: eval: epoch 116 | batch 0/512
[I 2022-10-28 13:17:07] ...trap/engines/engine.py.325:       elapsed: 0:00:02 | left: 0:24:23
[I 2022-10-28 13:17:07] ...trap/engines/engine.py.326:       process: 2.85847 | load: 1.54239
[I 2022-10-28 13:17:10] ...trap/engines/engine.py.321: eval: epoch 116 | batch 10/512
[I 2022-10-28 13:17:10] ...trap/engines/engine.py.325:       elapsed: 0:00:05 | left: 0:04:48
[I 2022-10-28 13:17:10] ...trap/engines/engine.py.326:       process: 0.29329 | load: 0.00015
[I 2022-10-28 13:17:13] ...trap/engines/engine.py.321: eval: epoch 116 | batch 20/512
[I 2022-10-28 13:17:13] ...trap/engines/engine.py.325:       elapsed: 0:00:08 | left: 0:02:41
[I 2022-10-28 13:17:13] ...trap/engines/engine.py.326:       process: 0.29574 | load: 0.00014
[I 2022-10-28 13:17:17] ...trap/engines/engine.py.321: eval: epoch 116 | batch 30/512
[I 2022-10-28 13:17:17] ...trap/engines/engine.py.325:       elapsed: 0:00:11 | left: 0:02:26
[I 2022-10-28 13:17:17] ...trap/engines/engine.py.326:       process: 0.30439 | load: 0.00014
[I 2022-10-28 13:17:20] ...trap/engines/engine.py.321: eval: epoch 116 | batch 40/512
[I 2022-10-28 13:17:20] ...trap/engines/engine.py.325:       elapsed: 0:00:14 | left: 0:02:22
[I 2022-10-28 13:17:20] ...trap/engines/engine.py.326:       process: 0.29752 | load: 0.00015
[I 2022-10-28 13:17:23] ...trap/engines/engine.py.321: eval: epoch 116 | batch 50/512
[I 2022-10-28 13:17:23] ...trap/engines/engine.py.325:       elapsed: 0:00:17 | left: 0:02:18
[I 2022-10-28 13:17:23] ...trap/engines/engine.py.326:       process: 0.31043 | load: 0.00017
[I 2022-10-28 13:17:26] ...trap/engines/engine.py.321: eval: epoch 116 | batch 60/512
[I 2022-10-28 13:17:26] ...trap/engines/engine.py.325:       elapsed: 0:00:21 | left: 0:02:16
[I 2022-10-28 13:17:26] ...trap/engines/engine.py.326:       process: 0.29547 | load: 0.00018
[I 2022-10-28 13:17:29] ...trap/engines/engine.py.321: eval: epoch 116 | batch 70/512
[I 2022-10-28 13:17:29] ...trap/engines/engine.py.325:       elapsed: 0:00:24 | left: 0:02:15
[I 2022-10-28 13:17:29] ...trap/engines/engine.py.326:       process: 0.33300 | load: 0.00018
[I 2022-10-28 13:17:32] ...trap/engines/engine.py.321: eval: epoch 116 | batch 80/512
[I 2022-10-28 13:17:32] ...trap/engines/engine.py.325:       elapsed: 0:00:27 | left: 0:02:08
[I 2022-10-28 13:17:32] ...trap/engines/engine.py.326:       process: 0.29864 | load: 0.00016
[I 2022-10-28 13:17:35] ...trap/engines/engine.py.321: eval: epoch 116 | batch 90/512
[I 2022-10-28 13:17:35] ...trap/engines/engine.py.325:       elapsed: 0:00:30 | left: 0:02:07
[I 2022-10-28 13:17:35] ...trap/engines/engine.py.326:       process: 0.29857 | load: 0.00015
[I 2022-10-28 13:17:38] ...trap/engines/engine.py.321: eval: epoch 116 | batch 100/512
[I 2022-10-28 13:17:38] ...trap/engines/engine.py.325:       elapsed: 0:00:33 | left: 0:01:59
[I 2022-10-28 13:17:38] ...trap/engines/engine.py.326:       process: 0.26042 | load: 0.00017
[I 2022-10-28 13:17:40] ...trap/engines/engine.py.321: eval: epoch 116 | batch 110/512
[I 2022-10-28 13:17:40] ...trap/engines/engine.py.325:       elapsed: 0:00:35 | left: 0:01:44
[I 2022-10-28 13:17:40] ...trap/engines/engine.py.326:       process: 0.25823 | load: 0.00019
[I 2022-10-28 13:17:43] ...trap/engines/engine.py.321: eval: epoch 116 | batch 120/512
[I 2022-10-28 13:17:43] ...trap/engines/engine.py.325:       elapsed: 0:00:38 | left: 0:01:41
[I 2022-10-28 13:17:43] ...trap/engines/engine.py.326:       process: 0.25824 | load: 0.00013
[I 2022-10-28 13:17:45] ...trap/engines/engine.py.321: eval: epoch 116 | batch 130/512
[I 2022-10-28 13:17:45] ...trap/engines/engine.py.325:       elapsed: 0:00:40 | left: 0:01:38
[I 2022-10-28 13:17:45] ...trap/engines/engine.py.326:       process: 0.25850 | load: 0.00014
[I 2022-10-28 13:17:48] ...trap/engines/engine.py.321: eval: epoch 116 | batch 140/512
[I 2022-10-28 13:17:48] ...trap/engines/engine.py.325:       elapsed: 0:00:43 | left: 0:01:35
[I 2022-10-28 13:17:48] ...trap/engines/engine.py.326:       process: 0.25465 | load: 0.00015
[I 2022-10-28 13:17:51] ...trap/engines/engine.py.321: eval: epoch 116 | batch 150/512
[I 2022-10-28 13:17:51] ...trap/engines/engine.py.325:       elapsed: 0:00:45 | left: 0:01:33
[I 2022-10-28 13:17:51] ...trap/engines/engine.py.326:       process: 0.25947 | load: 0.00017
[I 2022-10-28 13:17:53] ...trap/engines/engine.py.321: eval: epoch 116 | batch 160/512
[I 2022-10-28 13:17:53] ...trap/engines/engine.py.325:       elapsed: 0:00:48 | left: 0:01:29
[I 2022-10-28 13:17:53] ...trap/engines/engine.py.326:       process: 0.25614 | load: 0.00014
[I 2022-10-28 13:17:56] ...trap/engines/engine.py.321: eval: epoch 116 | batch 170/512
[I 2022-10-28 13:17:56] ...trap/engines/engine.py.325:       elapsed: 0:00:51 | left: 0:01:27
[I 2022-10-28 13:17:56] ...trap/engines/engine.py.326:       process: 0.25881 | load: 0.00014
[I 2022-10-28 13:17:58] ...trap/engines/engine.py.321: eval: epoch 116 | batch 180/512
[I 2022-10-28 13:17:58] ...trap/engines/engine.py.325:       elapsed: 0:00:53 | left: 0:01:24
[I 2022-10-28 13:17:58] ...trap/engines/engine.py.326:       process: 0.25412 | load: 0.00018
[I 2022-10-28 13:18:01] ...trap/engines/engine.py.321: eval: epoch 116 | batch 190/512
[I 2022-10-28 13:18:01] ...trap/engines/engine.py.325:       elapsed: 0:00:56 | left: 0:01:24
[I 2022-10-28 13:18:01] ...trap/engines/engine.py.326:       process: 0.27757 | load: 0.00017
[I 2022-10-28 13:18:03] ...trap/engines/engine.py.321: eval: epoch 116 | batch 200/512
[I 2022-10-28 13:18:03] ...trap/engines/engine.py.325:       elapsed: 0:00:58 | left: 0:01:20
[I 2022-10-28 13:18:03] ...trap/engines/engine.py.326:       process: 0.25881 | load: 0.00016
[I 2022-10-28 13:18:06] ...trap/engines/engine.py.321: eval: epoch 116 | batch 210/512
[I 2022-10-28 13:18:06] ...trap/engines/engine.py.325:       elapsed: 0:01:01 | left: 0:01:17
[I 2022-10-28 13:18:06] ...trap/engines/engine.py.326:       process: 0.25539 | load: 0.00017
[I 2022-10-28 13:18:08] ...trap/engines/engine.py.321: eval: epoch 116 | batch 220/512
[I 2022-10-28 13:18:08] ...trap/engines/engine.py.325:       elapsed: 0:01:03 | left: 0:01:14
[I 2022-10-28 13:18:08] ...trap/engines/engine.py.326:       process: 0.25231 | load: 0.00016
[I 2022-10-28 13:18:11] ...trap/engines/engine.py.321: eval: epoch 116 | batch 230/512
[I 2022-10-28 13:18:11] ...trap/engines/engine.py.325:       elapsed: 0:01:06 | left: 0:01:14
[I 2022-10-28 13:18:11] ...trap/engines/engine.py.326:       process: 0.27437 | load: 0.00014
[I 2022-10-28 13:18:14] ...trap/engines/engine.py.321: eval: epoch 116 | batch 240/512
[I 2022-10-28 13:18:14] ...trap/engines/engine.py.325:       elapsed: 0:01:09 | left: 0:01:10
[I 2022-10-28 13:18:14] ...trap/engines/engine.py.326:       process: 0.26415 | load: 0.00014
[I 2022-10-28 13:18:16] ...trap/engines/engine.py.321: eval: epoch 116 | batch 250/512
[I 2022-10-28 13:18:16] ...trap/engines/engine.py.325:       elapsed: 0:01:11 | left: 0:01:07
[I 2022-10-28 13:18:16] ...trap/engines/engine.py.326:       process: 0.25725 | load: 0.00015
[I 2022-10-28 13:18:19] ...trap/engines/engine.py.321: eval: epoch 116 | batch 260/512
[I 2022-10-28 13:18:19] ...trap/engines/engine.py.325:       elapsed: 0:01:14 | left: 0:01:04
[I 2022-10-28 13:18:19] ...trap/engines/engine.py.326:       process: 0.25629 | load: 0.00015
[I 2022-10-28 13:18:21] ...trap/engines/engine.py.321: eval: epoch 116 | batch 270/512
[I 2022-10-28 13:18:21] ...trap/engines/engine.py.325:       elapsed: 0:01:16 | left: 0:01:02
[I 2022-10-28 13:18:21] ...trap/engines/engine.py.326:       process: 0.26135 | load: 0.00015
[I 2022-10-28 13:18:24] ...trap/engines/engine.py.321: eval: epoch 116 | batch 280/512
[I 2022-10-28 13:18:24] ...trap/engines/engine.py.325:       elapsed: 0:01:19 | left: 0:00:59
[I 2022-10-28 13:18:24] ...trap/engines/engine.py.326:       process: 0.25733 | load: 0.00016
[I 2022-10-28 13:18:27] ...trap/engines/engine.py.321: eval: epoch 116 | batch 290/512
[I 2022-10-28 13:18:27] ...trap/engines/engine.py.325:       elapsed: 0:01:21 | left: 0:00:56
[I 2022-10-28 13:18:27] ...trap/engines/engine.py.326:       process: 0.24661 | load: 0.00018
[I 2022-10-28 13:18:29] ...trap/engines/engine.py.321: eval: epoch 116 | batch 300/512
[I 2022-10-28 13:18:29] ...trap/engines/engine.py.325:       elapsed: 0:01:24 | left: 0:00:54
[I 2022-10-28 13:18:29] ...trap/engines/engine.py.326:       process: 0.25574 | load: 0.00018
[I 2022-10-28 13:18:32] ...trap/engines/engine.py.321: eval: epoch 116 | batch 310/512
[I 2022-10-28 13:18:32] ...trap/engines/engine.py.325:       elapsed: 0:01:27 | left: 0:00:52
[I 2022-10-28 13:18:32] ...trap/engines/engine.py.326:       process: 0.26006 | load: 0.00014
[I 2022-10-28 13:18:34] ...trap/engines/engine.py.321: eval: epoch 116 | batch 320/512
[I 2022-10-28 13:18:34] ...trap/engines/engine.py.325:       elapsed: 0:01:29 | left: 0:00:49
[I 2022-10-28 13:18:34] ...trap/engines/engine.py.326:       process: 0.25366 | load: 0.00015
[I 2022-10-28 13:18:37] ...trap/engines/engine.py.321: eval: epoch 116 | batch 330/512
[I 2022-10-28 13:18:37] ...trap/engines/engine.py.325:       elapsed: 0:01:32 | left: 0:00:46
[I 2022-10-28 13:18:37] ...trap/engines/engine.py.326:       process: 0.27275 | load: 0.00014
[I 2022-10-28 13:18:39] ...trap/engines/engine.py.321: eval: epoch 116 | batch 340/512
[I 2022-10-28 13:18:39] ...trap/engines/engine.py.325:       elapsed: 0:01:34 | left: 0:00:44
[I 2022-10-28 13:18:39] ...trap/engines/engine.py.326:       process: 0.25895 | load: 0.00017
[I 2022-10-28 13:18:42] ...trap/engines/engine.py.321: eval: epoch 116 | batch 350/512
[I 2022-10-28 13:18:42] ...trap/engines/engine.py.325:       elapsed: 0:01:37 | left: 0:00:41
[I 2022-10-28 13:18:42] ...trap/engines/engine.py.326:       process: 0.25584 | load: 0.00014
[I 2022-10-28 13:18:45] ...trap/engines/engine.py.321: eval: epoch 116 | batch 360/512
[I 2022-10-28 13:18:45] ...trap/engines/engine.py.325:       elapsed: 0:01:40 | left: 0:00:39
[I 2022-10-28 13:18:45] ...trap/engines/engine.py.326:       process: 0.26099 | load: 0.00017
[I 2022-10-28 13:18:47] ...trap/engines/engine.py.321: eval: epoch 116 | batch 370/512
[I 2022-10-28 13:18:47] ...trap/engines/engine.py.325:       elapsed: 0:01:42 | left: 0:00:36
[I 2022-10-28 13:18:47] ...trap/engines/engine.py.326:       process: 0.27253 | load: 0.00015
[I 2022-10-28 13:18:50] ...trap/engines/engine.py.321: eval: epoch 116 | batch 380/512
[I 2022-10-28 13:18:50] ...trap/engines/engine.py.325:       elapsed: 0:01:45 | left: 0:00:33
[I 2022-10-28 13:18:50] ...trap/engines/engine.py.326:       process: 0.25866 | load: 0.00013
[I 2022-10-28 13:18:52] ...trap/engines/engine.py.321: eval: epoch 116 | batch 390/512
[I 2022-10-28 13:18:52] ...trap/engines/engine.py.325:       elapsed: 0:01:47 | left: 0:00:31
[I 2022-10-28 13:18:52] ...trap/engines/engine.py.326:       process: 0.25590 | load: 0.00016
[I 2022-10-28 13:18:55] ...trap/engines/engine.py.321: eval: epoch 116 | batch 400/512
[I 2022-10-28 13:18:55] ...trap/engines/engine.py.325:       elapsed: 0:01:50 | left: 0:00:28
[I 2022-10-28 13:18:55] ...trap/engines/engine.py.326:       process: 0.25485 | load: 0.00017
[I 2022-10-28 13:18:57] ...trap/engines/engine.py.321: eval: epoch 116 | batch 410/512
[I 2022-10-28 13:18:57] ...trap/engines/engine.py.325:       elapsed: 0:01:52 | left: 0:00:26
[I 2022-10-28 13:18:57] ...trap/engines/engine.py.326:       process: 0.25963 | load: 0.00016
[I 2022-10-28 13:19:00] ...trap/engines/engine.py.321: eval: epoch 116 | batch 420/512
[I 2022-10-28 13:19:00] ...trap/engines/engine.py.325:       elapsed: 0:01:55 | left: 0:00:23
[I 2022-10-28 13:19:00] ...trap/engines/engine.py.326:       process: 0.25736 | load: 0.00016
[I 2022-10-28 13:19:03] ...trap/engines/engine.py.321: eval: epoch 116 | batch 430/512
[I 2022-10-28 13:19:03] ...trap/engines/engine.py.325:       elapsed: 0:01:58 | left: 0:00:21
[I 2022-10-28 13:19:03] ...trap/engines/engine.py.326:       process: 0.25576 | load: 0.00016
[I 2022-10-28 13:19:05] ...trap/engines/engine.py.321: eval: epoch 116 | batch 440/512
[I 2022-10-28 13:19:05] ...trap/engines/engine.py.325:       elapsed: 0:02:00 | left: 0:00:18
[I 2022-10-28 13:19:05] ...trap/engines/engine.py.326:       process: 0.25631 | load: 0.00016
[I 2022-10-28 13:19:08] ...trap/engines/engine.py.321: eval: epoch 116 | batch 450/512
[I 2022-10-28 13:19:08] ...trap/engines/engine.py.325:       elapsed: 0:02:03 | left: 0:00:15
[I 2022-10-28 13:19:08] ...trap/engines/engine.py.326:       process: 0.25031 | load: 0.00016
[I 2022-10-28 13:19:10] ...trap/engines/engine.py.321: eval: epoch 116 | batch 460/512
[I 2022-10-28 13:19:10] ...trap/engines/engine.py.325:       elapsed: 0:02:05 | left: 0:00:13
[I 2022-10-28 13:19:10] ...trap/engines/engine.py.326:       process: 0.25494 | load: 0.00015
[I 2022-10-28 13:19:13] ...trap/engines/engine.py.321: eval: epoch 116 | batch 470/512
[I 2022-10-28 13:19:13] ...trap/engines/engine.py.325:       elapsed: 0:02:08 | left: 0:00:10
[I 2022-10-28 13:19:13] ...trap/engines/engine.py.326:       process: 0.24921 | load: 0.00017
[I 2022-10-28 13:19:15] ...trap/engines/engine.py.321: eval: epoch 116 | batch 480/512
[I 2022-10-28 13:19:15] ...trap/engines/engine.py.325:       elapsed: 0:02:10 | left: 0:00:08
[I 2022-10-28 13:19:15] ...trap/engines/engine.py.326:       process: 0.25572 | load: 0.00016
[I 2022-10-28 13:19:18] ...trap/engines/engine.py.321: eval: epoch 116 | batch 490/512
[I 2022-10-28 13:19:18] ...trap/engines/engine.py.325:       elapsed: 0:02:13 | left: 0:00:05
[I 2022-10-28 13:19:18] ...trap/engines/engine.py.326:       process: 0.24972 | load: 0.00016
[I 2022-10-28 13:19:21] ...trap/engines/engine.py.321: eval: epoch 116 | batch 500/512
[I 2022-10-28 13:19:21] ...trap/engines/engine.py.325:       elapsed: 0:02:15 | left: 0:00:03
[I 2022-10-28 13:19:21] ...trap/engines/engine.py.326:       process: 0.25448 | load: 0.00016
[I 2022-10-28 13:19:23] ...trap/engines/engine.py.321: eval: epoch 116 | batch 510/512
[I 2022-10-28 13:19:23] ...trap/engines/engine.py.325:       elapsed: 0:02:18 | left: 0:00:00
[I 2022-10-28 13:19:23] ...trap/engines/engine.py.326:       process: 0.25078 | load: 0.00011
[S 2022-10-28 13:19:24] ...trap/engines/engine.py.113: eval_epoch.epoch: 116
[I 2022-10-28 13:19:24] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs
[S 2022-10-28 13:19:28] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00
[S 2022-10-28 13:19:28] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00
[S 2022-10-28 13:19:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.620100
[S 2022-10-28 13:19:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.837500
[S 2022-10-28 13:19:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.885800
[S 2022-10-28 13:19:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.641000
[S 2022-10-28 13:19:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.857100
[S 2022-10-28 13:19:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.900400
[S 2022-10-28 13:19:28] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000
[S 2022-10-28 13:19:28] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000
[S 2022-10-28 13:19:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.014604
[S 2022-10-28 13:19:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.011944
[S 2022-10-28 13:19:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.008424
[S 2022-10-28 13:19:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.013993
[S 2022-10-28 13:19:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.008154
[S 2022-10-28 13:19:28] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.006168
[I 2022-10-28 13:19:29] ...trap/engines/engine.py.115: Ending evaluation procedures
[W 2022-10-28 13:19:29] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json'
[W 2022-10-28 13:19:29] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[W 2022-10-28 13:19:29] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[I 2022-10-28 13:19:33] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html
[I 2022-10-28 13:21:36] ...ytorch/bootstrap/run.py.73: Saving environment info
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75: options: 
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:   dataset:
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     aux_kw_path:
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       test: /home/mshukor/data/recipe1m/clip_da/layer1_test_titles_kw.json
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       train: /home/mshukor/data/recipe1m/clip_da/layer1_train_titles_kw.json
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       val: /home/mshukor/data/recipe1m/clip_da/layer1_val_titles_kw.json
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     batch_sampler: triplet_classif
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     batch_size: 100
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     database: lmdb
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     dir: /home/mshukor/data/recipe1m
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     eval_split: test
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     freq_mismatch: 0.0
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     image_from: database
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     import: recipe1m.datasets.factory
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     interchange_ingrd_instr: None
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     kw_path:
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       test: /home/mshukor/data/recipe1m/clip_da/layer1_test_ingr_kw.json
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       train: /home/mshukor/data/recipe1m/clip_da/layer1_train_ingr_kw.json
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       val: /home/mshukor/data/recipe1m/clip_da/layer1_val_ingr_kw.json
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     name: recipe1m
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     nb_classes: 1048
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     nb_threads: 4
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     randkw_p: 0.3
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     randkw_p_aux: 0.5
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     remove_list: [ingredients]
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     tokenized_raw_text: True
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     train_split: None
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     use_vcs: True
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:   engine:
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     name: logger
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     nb_epochs: 120
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     print_freq: 10
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     saving_criteria: ['eval_epoch.metric.med_im2recipe_mean:min', 'eval_epoch.metric.recall_at_1_im2recipe_mean:max']
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:   exp:
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     checkpoint: None
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     dir: /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     resume: best_eval_epoch.metric.recall_at_1_im2recipe_mean
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:   misc:
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     cuda: True
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     data_parrallel: True
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     logs_name: None
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     overrite: False
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     seed: 1338
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:   model:
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     criterion:
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:         tri_cross_encoder: False
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       itm_loss_weight: 1
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       keep_background: False
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       retrieval_strategy:
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:         aggregation: valid
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:         get_tokens: True
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:         margin: 0.05
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:         margin_params:
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:           increment: 0.005
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:           increment_margin: True
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:           max_margin: 0.3
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:         name: triplet
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:         nb_samples: 9999
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:         sampling: max_negative
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:         substrategy: ['IRR', 'RII', 'SIRR', 'SRII']
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:         substrategy_weights: [1.0, 1.0, 0.1, 0.1]
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     import: recipe1m.models.factory
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     metric:
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       k_test: 10
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       nb_bags: 10
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       nb_matchs_per_bag: 1000
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       retrieval_dir: /tmp/recipe1m
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       save_ids: False
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       trijoint: True
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     name: trijoint
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     network:
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       activations: ['tanh', 'normalize']
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       aux_kwords: True
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       aux_kwords_encoder: True
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       avg_concat: True
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       bert_config: /home/mshukor/tfood/recipe1m/models/networks/recipe_networks/config_bert_albef.json
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       cat_pos: 0
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       checkpoint: None
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       class_attention: False
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       class_attention_fus: False
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       cls_token: False
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       cls_token_fus: False
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       cross_decoder_image: True
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       cross_decoder_recipe: True
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:         class_attention: False
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:         cls_token: False
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:         cross_attention: False
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:         cross_decoder: True
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:         double_decoder: False
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:         get_tokens: False
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:         load_backbones: False
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:         n_heads: 4
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:         n_layers: 4
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:         query_img: False
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       cross_transformer: False
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       dim_emb: 1024
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       dim_image_out: 2048
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       dim_ingr_out: 300
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       dim_instr_in: 1024
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       dim_instr_out: 1024
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       freeze_im: False
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       freeze_rec: False
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       get_tokens: True
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       get_tokens_cross_decoder_recipe: True
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       hidden_size: 512
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       image_backbone_name: clip
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       image_res: 224
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       kwords_same_level: False
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       last_cat_aux: True
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       n_heads: 4
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       n_heads_cross: 4
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       n_heads_cross_image: 4
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       n_heads_single: 4
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       n_layers: 2
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       n_layers_cross: 2
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       n_layers_cross_image: 1
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       n_layers_single: 2
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       num_hidden_layers_kw: 2
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       old: True
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       path_ingrs: /home/mshukor/data/recipe1m/text/vocab.pkl
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       path_vocab: /home/mshukor/data/recipe1m/text/vocab_all.txt
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       recipe_encoder: tfoodvicha-transformer
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       remove_additional_embedding: ['proj_image']
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       text_encoder: bert-base-uncased
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       vision_width: 768
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       vit_all_tokens: True
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       with_ingrs: True
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       with_instrs: True
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:       with_titles: True
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     with_classif: False
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:   optimizer:
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     batch_size_factor: None
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     clip_grad: 8.0
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     import: recipe1m.optimizers.factory
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     lr: 1e-05
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     lr_img: 1e-06
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     name: trijoint_fixed_fine_tune
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:     switch_epoch: 20
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:   path_opts: /home/mshukor/tfood/recipe1m/options/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_noimgproj.yaml
[S 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.75:   view: ['logs:train_epoch.loss', 'logs:train_epoch.bad_pairs', 'logs:eval_epoch.metric.med_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_1_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_5_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_10_im2recipe_mean', 'logs:eval_epoch.metric.med_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_1_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_5_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_10_recipe2im_mean', 'logs:optimizer.is_optimizer_recipe&image', 'logs:optimizer.total_norm']
[I 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.76: posix.uname_result(sysname='Linux', nodename='DGX', release='5.4.0-124-generic', version='#140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022', machine='x86_64')
[I 2022-10-28 13:21:37] ...ytorch/bootstrap/run.py.80: Available GPUs: [0]
[I 2022-10-28 13:21:37] ...strap/engines/factory.py.9: Creating engine...
[I 2022-10-28 13:21:37] ...trap/datasets/factory.py.8: Creating dataset...
[I 2022-10-28 13:21:39] ...m/datasets/recipe1m.py.152: Load VCs...
[I 2022-10-28 13:21:39] ...m/datasets/recipe1m.py.165: randkw_p... None
[I 2022-10-28 13:22:42] ...m/datasets/recipe1m.py.310: recipe elements to remove: [ingredients]
[I 2022-10-28 13:22:42] ...rap/datasets/factory.py.21: Evaluation will take place on testset (51303 items)
[I 2022-10-28 13:22:42] ...strap/models/factory.py.21: Creating model...
[I 2022-10-28 13:22:43] ...e_networks/networks.py.203: Loading CLIP Visual model...
[I 2022-10-28 13:22:46] ...e_networks/networks.py.145: cuda
[I 2022-10-28 13:22:48] ...networks/tfood_vicha.py.49: Load VCs encoder...
[I 2022-10-28 13:22:50] ...els/metrics/trijoint.py.46: self.save_ids False
[I 2022-10-28 13:22:50] ...strap/models/factory.py.44: Enabling CUDA mode...
[I 2022-10-28 13:22:52] ...etworks/tfood_vicha.py.184: training aux kw modules
[I 2022-10-28 13:22:52] ...etworks/tfood_vicha.py.184: training aux kw modules
[I 2022-10-28 13:22:52] ...otstrap/views/factory.py.8: Creating views...
[I 2022-10-28 13:22:53] ...strap/engines/engine.py.98: Loading best_eval_epoch.metric.recall_at_1_im2recipe_mean checkpoint
[I 2022-10-28 13:22:53] ...trap/engines/engine.py.440: Loading model...
[I 2022-10-28 13:22:53] ...trap/engines/engine.py.453: Loading engine...
[I 2022-10-28 13:22:53] ...trap/engines/engine.py.108: Launching evaluation procedures
[I 2022-10-28 13:22:53] ...trap/engines/engine.py.268: Evaluating model on testset for epoch 116
[I 2022-10-28 13:22:53] ...m/datasets/recipe1m.py.400: Dataset will be sampled with "random" batch_sampler.
[I 2022-10-28 13:22:56] ...trap/engines/engine.py.321: eval: epoch 116 | batch 0/512
[I 2022-10-28 13:22:56] ...trap/engines/engine.py.325:       elapsed: 0:00:02 | left: 0:23:03
[I 2022-10-28 13:22:56] ...trap/engines/engine.py.326:       process: 2.70249 | load: 1.49259
[I 2022-10-28 13:22:59] ...trap/engines/engine.py.321: eval: epoch 116 | batch 10/512
[I 2022-10-28 13:22:59] ...trap/engines/engine.py.325:       elapsed: 0:00:05 | left: 0:04:41
[I 2022-10-28 13:22:59] ...trap/engines/engine.py.326:       process: 0.30192 | load: 0.00015
[I 2022-10-28 13:23:02] ...trap/engines/engine.py.321: eval: epoch 116 | batch 20/512
[I 2022-10-28 13:23:02] ...trap/engines/engine.py.325:       elapsed: 0:00:08 | left: 0:02:41
[I 2022-10-28 13:23:02] ...trap/engines/engine.py.326:       process: 0.29733 | load: 0.00015
[I 2022-10-28 13:23:05] ...trap/engines/engine.py.321: eval: epoch 116 | batch 30/512
[I 2022-10-28 13:23:05] ...trap/engines/engine.py.325:       elapsed: 0:00:11 | left: 0:02:28
[I 2022-10-28 13:23:05] ...trap/engines/engine.py.326:       process: 0.29998 | load: 0.00018
[I 2022-10-28 13:23:08] ...trap/engines/engine.py.321: eval: epoch 116 | batch 40/512
[I 2022-10-28 13:23:08] ...trap/engines/engine.py.325:       elapsed: 0:00:14 | left: 0:02:21
[I 2022-10-28 13:23:08] ...trap/engines/engine.py.326:       process: 0.29591 | load: 0.00015
[I 2022-10-28 13:23:11] ...trap/engines/engine.py.321: eval: epoch 116 | batch 50/512
[I 2022-10-28 13:23:11] ...trap/engines/engine.py.325:       elapsed: 0:00:17 | left: 0:02:18
[I 2022-10-28 13:23:11] ...trap/engines/engine.py.326:       process: 0.30076 | load: 0.00017
[I 2022-10-28 13:23:14] ...trap/engines/engine.py.321: eval: epoch 116 | batch 60/512
[I 2022-10-28 13:23:14] ...trap/engines/engine.py.325:       elapsed: 0:00:20 | left: 0:02:18
[I 2022-10-28 13:23:14] ...trap/engines/engine.py.326:       process: 0.32501 | load: 0.00015
[I 2022-10-28 13:23:17] ...trap/engines/engine.py.321: eval: epoch 116 | batch 70/512
[I 2022-10-28 13:23:17] ...trap/engines/engine.py.325:       elapsed: 0:00:24 | left: 0:02:18
[I 2022-10-28 13:23:17] ...trap/engines/engine.py.326:       process: 0.30361 | load: 0.00016
[I 2022-10-28 13:23:21] ...trap/engines/engine.py.321: eval: epoch 116 | batch 80/512
[I 2022-10-28 13:23:21] ...trap/engines/engine.py.325:       elapsed: 0:00:27 | left: 0:02:12
[I 2022-10-28 13:23:21] ...trap/engines/engine.py.326:       process: 0.32173 | load: 0.00016
[I 2022-10-28 13:23:24] ...trap/engines/engine.py.321: eval: epoch 116 | batch 90/512
[I 2022-10-28 13:23:24] ...trap/engines/engine.py.325:       elapsed: 0:00:30 | left: 0:02:06
[I 2022-10-28 13:23:24] ...trap/engines/engine.py.326:       process: 0.29920 | load: 0.00019
[I 2022-10-28 13:23:27] ...trap/engines/engine.py.321: eval: epoch 116 | batch 100/512
[I 2022-10-28 13:23:27] ...trap/engines/engine.py.325:       elapsed: 0:00:33 | left: 0:02:02
[I 2022-10-28 13:23:27] ...trap/engines/engine.py.326:       process: 0.26202 | load: 0.00015
[I 2022-10-28 13:23:29] ...trap/engines/engine.py.321: eval: epoch 116 | batch 110/512
[I 2022-10-28 13:23:29] ...trap/engines/engine.py.325:       elapsed: 0:00:35 | left: 0:01:46
[I 2022-10-28 13:23:29] ...trap/engines/engine.py.326:       process: 0.26528 | load: 0.00014
[I 2022-10-28 13:23:32] ...trap/engines/engine.py.321: eval: epoch 116 | batch 120/512
[I 2022-10-28 13:23:32] ...trap/engines/engine.py.325:       elapsed: 0:00:38 | left: 0:01:42
[I 2022-10-28 13:23:32] ...trap/engines/engine.py.326:       process: 0.26245 | load: 0.00014
[I 2022-10-28 13:23:34] ...trap/engines/engine.py.321: eval: epoch 116 | batch 130/512
[I 2022-10-28 13:23:34] ...trap/engines/engine.py.325:       elapsed: 0:00:40 | left: 0:01:40
[I 2022-10-28 13:23:34] ...trap/engines/engine.py.326:       process: 0.26326 | load: 0.00014
[I 2022-10-28 13:23:37] ...trap/engines/engine.py.321: eval: epoch 116 | batch 140/512
[I 2022-10-28 13:23:37] ...trap/engines/engine.py.325:       elapsed: 0:00:43 | left: 0:01:37
[I 2022-10-28 13:23:37] ...trap/engines/engine.py.326:       process: 0.25813 | load: 0.00018
[I 2022-10-28 13:23:40] ...trap/engines/engine.py.321: eval: epoch 116 | batch 150/512
[I 2022-10-28 13:23:40] ...trap/engines/engine.py.325:       elapsed: 0:00:46 | left: 0:01:35
[I 2022-10-28 13:23:40] ...trap/engines/engine.py.326:       process: 0.26116 | load: 0.00017
[I 2022-10-28 13:23:42] ...trap/engines/engine.py.321: eval: epoch 116 | batch 160/512
[I 2022-10-28 13:23:42] ...trap/engines/engine.py.325:       elapsed: 0:00:48 | left: 0:01:33
[I 2022-10-28 13:23:42] ...trap/engines/engine.py.326:       process: 0.26291 | load: 0.00015
[I 2022-10-28 13:23:45] ...trap/engines/engine.py.321: eval: epoch 116 | batch 170/512
[I 2022-10-28 13:23:45] ...trap/engines/engine.py.325:       elapsed: 0:00:51 | left: 0:01:29
[I 2022-10-28 13:23:45] ...trap/engines/engine.py.326:       process: 0.26697 | load: 0.00014
[I 2022-10-28 13:23:48] ...trap/engines/engine.py.321: eval: epoch 116 | batch 180/512
[I 2022-10-28 13:23:48] ...trap/engines/engine.py.325:       elapsed: 0:00:54 | left: 0:01:26
[I 2022-10-28 13:23:48] ...trap/engines/engine.py.326:       process: 0.25929 | load: 0.00015
[I 2022-10-28 13:23:50] ...trap/engines/engine.py.321: eval: epoch 116 | batch 190/512
[I 2022-10-28 13:23:50] ...trap/engines/engine.py.325:       elapsed: 0:00:56 | left: 0:01:24
[I 2022-10-28 13:23:50] ...trap/engines/engine.py.326:       process: 0.26509 | load: 0.00014
[I 2022-10-28 13:23:53] ...trap/engines/engine.py.321: eval: epoch 116 | batch 200/512
[I 2022-10-28 13:23:53] ...trap/engines/engine.py.325:       elapsed: 0:00:59 | left: 0:01:21
[I 2022-10-28 13:23:53] ...trap/engines/engine.py.326:       process: 0.26403 | load: 0.00015
[I 2022-10-28 13:23:55] ...trap/engines/engine.py.321: eval: epoch 116 | batch 210/512
[I 2022-10-28 13:23:55] ...trap/engines/engine.py.325:       elapsed: 0:01:01 | left: 0:01:19
[I 2022-10-28 13:23:55] ...trap/engines/engine.py.326:       process: 0.27548 | load: 0.00014
[I 2022-10-28 13:23:58] ...trap/engines/engine.py.321: eval: epoch 116 | batch 220/512
[I 2022-10-28 13:23:58] ...trap/engines/engine.py.325:       elapsed: 0:01:04 | left: 0:01:16
[I 2022-10-28 13:23:58] ...trap/engines/engine.py.326:       process: 0.25955 | load: 0.00018
[I 2022-10-28 13:24:01] ...trap/engines/engine.py.321: eval: epoch 116 | batch 230/512
[I 2022-10-28 13:24:01] ...trap/engines/engine.py.325:       elapsed: 0:01:07 | left: 0:01:13
[I 2022-10-28 13:24:01] ...trap/engines/engine.py.326:       process: 0.26178 | load: 0.00014
[I 2022-10-28 13:24:03] ...trap/engines/engine.py.321: eval: epoch 116 | batch 240/512
[I 2022-10-28 13:24:03] ...trap/engines/engine.py.325:       elapsed: 0:01:09 | left: 0:01:11
[I 2022-10-28 13:24:03] ...trap/engines/engine.py.326:       process: 0.25877 | load: 0.00014
[I 2022-10-28 13:24:06] ...trap/engines/engine.py.321: eval: epoch 116 | batch 250/512
[I 2022-10-28 13:24:06] ...trap/engines/engine.py.325:       elapsed: 0:01:12 | left: 0:01:08
[I 2022-10-28 13:24:06] ...trap/engines/engine.py.326:       process: 0.26319 | load: 0.00014
[I 2022-10-28 13:24:08] ...trap/engines/engine.py.321: eval: epoch 116 | batch 260/512
[I 2022-10-28 13:24:08] ...trap/engines/engine.py.325:       elapsed: 0:01:14 | left: 0:01:05
[I 2022-10-28 13:24:08] ...trap/engines/engine.py.326:       process: 0.25934 | load: 0.00018
[I 2022-10-28 13:24:11] ...trap/engines/engine.py.321: eval: epoch 116 | batch 270/512
[I 2022-10-28 13:24:11] ...trap/engines/engine.py.325:       elapsed: 0:01:17 | left: 0:01:03
[I 2022-10-28 13:24:11] ...trap/engines/engine.py.326:       process: 0.26583 | load: 0.00015
[I 2022-10-28 13:24:14] ...trap/engines/engine.py.321: eval: epoch 116 | batch 280/512
[I 2022-10-28 13:24:14] ...trap/engines/engine.py.325:       elapsed: 0:01:20 | left: 0:01:00
[I 2022-10-28 13:24:14] ...trap/engines/engine.py.326:       process: 0.25825 | load: 0.00015
[I 2022-10-28 13:24:16] ...trap/engines/engine.py.321: eval: epoch 116 | batch 290/512
[I 2022-10-28 13:24:16] ...trap/engines/engine.py.325:       elapsed: 0:01:22 | left: 0:00:57
[I 2022-10-28 13:24:16] ...trap/engines/engine.py.326:       process: 0.26199 | load: 0.00015
[I 2022-10-28 13:24:19] ...trap/engines/engine.py.321: eval: epoch 116 | batch 300/512
[I 2022-10-28 13:24:19] ...trap/engines/engine.py.325:       elapsed: 0:01:25 | left: 0:00:55
[I 2022-10-28 13:24:19] ...trap/engines/engine.py.326:       process: 0.25686 | load: 0.00015
[I 2022-10-28 13:24:22] ...trap/engines/engine.py.321: eval: epoch 116 | batch 310/512
[I 2022-10-28 13:24:22] ...trap/engines/engine.py.325:       elapsed: 0:01:28 | left: 0:00:52
[I 2022-10-28 13:24:22] ...trap/engines/engine.py.326:       process: 0.26404 | load: 0.00015
[I 2022-10-28 13:24:24] ...trap/engines/engine.py.321: eval: epoch 116 | batch 320/512
[I 2022-10-28 13:24:24] ...trap/engines/engine.py.325:       elapsed: 0:01:30 | left: 0:00:50
[I 2022-10-28 13:24:24] ...trap/engines/engine.py.326:       process: 0.26138 | load: 0.00014
[I 2022-10-28 13:24:27] ...trap/engines/engine.py.321: eval: epoch 116 | batch 330/512
[I 2022-10-28 13:24:27] ...trap/engines/engine.py.325:       elapsed: 0:01:33 | left: 0:00:47
[I 2022-10-28 13:24:27] ...trap/engines/engine.py.326:       process: 0.25902 | load: 0.00015
[I 2022-10-28 13:24:29] ...trap/engines/engine.py.321: eval: epoch 116 | batch 340/512
[I 2022-10-28 13:24:29] ...trap/engines/engine.py.325:       elapsed: 0:01:35 | left: 0:00:45
[I 2022-10-28 13:24:29] ...trap/engines/engine.py.326:       process: 0.26228 | load: 0.00014
[I 2022-10-28 13:24:32] ...trap/engines/engine.py.321: eval: epoch 116 | batch 350/512
[I 2022-10-28 13:24:32] ...trap/engines/engine.py.325:       elapsed: 0:01:38 | left: 0:00:42
[I 2022-10-28 13:24:32] ...trap/engines/engine.py.326:       process: 0.26473 | load: 0.00017
[I 2022-10-28 13:24:35] ...trap/engines/engine.py.321: eval: epoch 116 | batch 360/512
[I 2022-10-28 13:24:35] ...trap/engines/engine.py.325:       elapsed: 0:01:41 | left: 0:00:39
[I 2022-10-28 13:24:35] ...trap/engines/engine.py.326:       process: 0.25915 | load: 0.00015
[I 2022-10-28 13:24:37] ...trap/engines/engine.py.321: eval: epoch 116 | batch 370/512
[I 2022-10-28 13:24:37] ...trap/engines/engine.py.325:       elapsed: 0:01:43 | left: 0:00:37
[I 2022-10-28 13:24:37] ...trap/engines/engine.py.326:       process: 0.26152 | load: 0.00016
[I 2022-10-28 13:24:40] ...trap/engines/engine.py.321: eval: epoch 116 | batch 380/512
[I 2022-10-28 13:24:40] ...trap/engines/engine.py.325:       elapsed: 0:01:46 | left: 0:00:34
[I 2022-10-28 13:24:40] ...trap/engines/engine.py.326:       process: 0.26242 | load: 0.00015
[I 2022-10-28 13:24:42] ...trap/engines/engine.py.321: eval: epoch 116 | batch 390/512
[I 2022-10-28 13:24:42] ...trap/engines/engine.py.325:       elapsed: 0:01:48 | left: 0:00:31
[I 2022-10-28 13:24:42] ...trap/engines/engine.py.326:       process: 0.26158 | load: 0.00018
[I 2022-10-28 13:24:45] ...trap/engines/engine.py.321: eval: epoch 116 | batch 400/512
[I 2022-10-28 13:24:45] ...trap/engines/engine.py.325:       elapsed: 0:01:51 | left: 0:00:29
[I 2022-10-28 13:24:45] ...trap/engines/engine.py.326:       process: 0.26295 | load: 0.00018
[I 2022-10-28 13:24:48] ...trap/engines/engine.py.321: eval: epoch 116 | batch 410/512
[I 2022-10-28 13:24:48] ...trap/engines/engine.py.325:       elapsed: 0:01:54 | left: 0:00:26
[I 2022-10-28 13:24:48] ...trap/engines/engine.py.326:       process: 0.26257 | load: 0.00018
[I 2022-10-28 13:24:50] ...trap/engines/engine.py.321: eval: epoch 116 | batch 420/512
[I 2022-10-28 13:24:50] ...trap/engines/engine.py.325:       elapsed: 0:01:56 | left: 0:00:23
[I 2022-10-28 13:24:50] ...trap/engines/engine.py.326:       process: 0.26080 | load: 0.00023
[I 2022-10-28 13:24:53] ...trap/engines/engine.py.321: eval: epoch 116 | batch 430/512
[I 2022-10-28 13:24:53] ...trap/engines/engine.py.325:       elapsed: 0:01:59 | left: 0:00:21
[I 2022-10-28 13:24:53] ...trap/engines/engine.py.326:       process: 0.26125 | load: 0.00014
[I 2022-10-28 13:24:55] ...trap/engines/engine.py.321: eval: epoch 116 | batch 440/512
[I 2022-10-28 13:24:55] ...trap/engines/engine.py.325:       elapsed: 0:02:01 | left: 0:00:18
[I 2022-10-28 13:24:55] ...trap/engines/engine.py.326:       process: 0.26395 | load: 0.00015
[I 2022-10-28 13:24:58] ...trap/engines/engine.py.321: eval: epoch 116 | batch 450/512
[I 2022-10-28 13:24:58] ...trap/engines/engine.py.325:       elapsed: 0:02:04 | left: 0:00:16
[I 2022-10-28 13:24:58] ...trap/engines/engine.py.326:       process: 0.26051 | load: 0.00015
[I 2022-10-28 13:25:01] ...trap/engines/engine.py.321: eval: epoch 116 | batch 460/512
[I 2022-10-28 13:25:01] ...trap/engines/engine.py.325:       elapsed: 0:02:07 | left: 0:00:13
[I 2022-10-28 13:25:01] ...trap/engines/engine.py.326:       process: 0.26162 | load: 0.00014
[I 2022-10-28 13:25:03] ...trap/engines/engine.py.321: eval: epoch 116 | batch 470/512
[I 2022-10-28 13:25:03] ...trap/engines/engine.py.325:       elapsed: 0:02:09 | left: 0:00:10
[I 2022-10-28 13:25:03] ...trap/engines/engine.py.326:       process: 0.26456 | load: 0.00015
[I 2022-10-28 13:25:06] ...trap/engines/engine.py.321: eval: epoch 116 | batch 480/512
[I 2022-10-28 13:25:06] ...trap/engines/engine.py.325:       elapsed: 0:02:12 | left: 0:00:08
[I 2022-10-28 13:25:06] ...trap/engines/engine.py.326:       process: 0.26080 | load: 0.00015
[I 2022-10-28 13:25:08] ...trap/engines/engine.py.321: eval: epoch 116 | batch 490/512
[I 2022-10-28 13:25:08] ...trap/engines/engine.py.325:       elapsed: 0:02:14 | left: 0:00:05
[I 2022-10-28 13:25:08] ...trap/engines/engine.py.326:       process: 0.26276 | load: 0.00016
[I 2022-10-28 13:25:11] ...trap/engines/engine.py.321: eval: epoch 116 | batch 500/512
[I 2022-10-28 13:25:11] ...trap/engines/engine.py.325:       elapsed: 0:02:17 | left: 0:00:03
[I 2022-10-28 13:25:11] ...trap/engines/engine.py.326:       process: 0.25958 | load: 0.00017
[I 2022-10-28 13:25:14] ...trap/engines/engine.py.321: eval: epoch 116 | batch 510/512
[I 2022-10-28 13:25:14] ...trap/engines/engine.py.325:       elapsed: 0:02:20 | left: 0:00:00
[I 2022-10-28 13:25:14] ...trap/engines/engine.py.326:       process: 0.25904 | load: 0.00012
[S 2022-10-28 13:25:15] ...trap/engines/engine.py.113: eval_epoch.epoch: 116
[I 2022-10-28 13:25:15] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs
[S 2022-10-28 13:25:19] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 4.80
[S 2022-10-28 13:25:19] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 2.40
[S 2022-10-28 13:25:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.249300
[S 2022-10-28 13:25:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.524300
[S 2022-10-28 13:25:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.639600
[S 2022-10-28 13:25:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.364800
[S 2022-10-28 13:25:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.657300
[S 2022-10-28 13:25:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.753300
[S 2022-10-28 13:25:19] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.400000
[S 2022-10-28 13:25:19] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.489898
[S 2022-10-28 13:25:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.012554
[S 2022-10-28 13:25:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.013682
[S 2022-10-28 13:25:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.010356
[S 2022-10-28 13:25:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.012311
[S 2022-10-28 13:25:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.012458
[S 2022-10-28 13:25:19] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.012673
[I 2022-10-28 13:25:19] ...trap/engines/engine.py.115: Ending evaluation procedures
[W 2022-10-28 13:25:20] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json'
[W 2022-10-28 13:25:20] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[W 2022-10-28 13:25:20] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[I 2022-10-28 13:25:24] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html
[I 2022-10-28 13:30:05] ...ytorch/bootstrap/run.py.73: Saving environment info
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75: options: 
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:   dataset:
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     aux_kw_path:
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       test: /home/mshukor/data/recipe1m/clip_da/layer1_test_titles_kw.json
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       train: /home/mshukor/data/recipe1m/clip_da/layer1_train_titles_kw.json
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       val: /home/mshukor/data/recipe1m/clip_da/layer1_val_titles_kw.json
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     batch_sampler: triplet_classif
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     batch_size: 100
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     database: lmdb
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     dir: /home/mshukor/data/recipe1m
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     eval_split: test
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     freq_mismatch: 0.0
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     image_from: database
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     import: recipe1m.datasets.factory
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     interchange_ingrd_instr: None
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     kw_path:
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       test: /home/mshukor/data/recipe1m/clip_da/layer1_test_ingr_kw.json
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       train: /home/mshukor/data/recipe1m/clip_da/layer1_train_ingr_kw.json
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       val: /home/mshukor/data/recipe1m/clip_da/layer1_val_ingr_kw.json
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     name: recipe1m
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     nb_classes: 1048
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     nb_threads: 4
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     randkw_p: 0.3
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     randkw_p_aux: 0.5
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     remove_list: [title]
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     tokenized_raw_text: True
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     train_split: None
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     use_vcs: True
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:   engine:
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     name: logger
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     nb_epochs: 120
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     print_freq: 10
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     saving_criteria: ['eval_epoch.metric.med_im2recipe_mean:min', 'eval_epoch.metric.recall_at_1_im2recipe_mean:max']
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:   exp:
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     checkpoint: None
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     dir: /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     resume: best_eval_epoch.metric.recall_at_1_im2recipe_mean
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:   misc:
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     cuda: True
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     data_parrallel: True
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     logs_name: None
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     overrite: False
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     seed: 1338
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:   model:
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     criterion:
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:         tri_cross_encoder: False
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       itm_loss_weight: 1
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       keep_background: False
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       retrieval_strategy:
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:         aggregation: valid
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:         get_tokens: True
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:         margin: 0.05
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:         margin_params:
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:           increment: 0.005
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:           increment_margin: True
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:           max_margin: 0.3
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:         name: triplet
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:         nb_samples: 9999
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:         sampling: max_negative
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:         substrategy: ['IRR', 'RII', 'SIRR', 'SRII']
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:         substrategy_weights: [1.0, 1.0, 0.1, 0.1]
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     import: recipe1m.models.factory
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     metric:
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       k_test: 10
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       nb_bags: 10
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       nb_matchs_per_bag: 1000
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       retrieval_dir: /tmp/recipe1m
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       save_ids: False
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       trijoint: True
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     name: trijoint
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     network:
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       activations: ['tanh', 'normalize']
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       aux_kwords: True
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       aux_kwords_encoder: True
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       avg_concat: True
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       bert_config: /home/mshukor/tfood/recipe1m/models/networks/recipe_networks/config_bert_albef.json
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       cat_pos: 0
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       checkpoint: None
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       class_attention: False
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       class_attention_fus: False
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       cls_token: False
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       cls_token_fus: False
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       cross_decoder_image: True
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       cross_decoder_recipe: True
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:         class_attention: False
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:         cls_token: False
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:         cross_attention: False
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:         cross_decoder: True
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:         double_decoder: False
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:         get_tokens: False
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:         load_backbones: False
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:         n_heads: 4
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:         n_layers: 4
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:         query_img: False
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       cross_transformer: False
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       dim_emb: 1024
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       dim_image_out: 2048
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       dim_ingr_out: 300
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       dim_instr_in: 1024
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       dim_instr_out: 1024
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       freeze_im: False
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       freeze_rec: False
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       get_tokens: True
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       get_tokens_cross_decoder_recipe: True
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       hidden_size: 512
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       image_backbone_name: clip
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       image_res: 224
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       kwords_same_level: False
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       last_cat_aux: True
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       n_heads: 4
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       n_heads_cross: 4
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       n_heads_cross_image: 4
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       n_heads_single: 4
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       n_layers: 2
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       n_layers_cross: 2
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       n_layers_cross_image: 1
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       n_layers_single: 2
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       num_hidden_layers_kw: 2
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       old: True
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       path_ingrs: /home/mshukor/data/recipe1m/text/vocab.pkl
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       path_vocab: /home/mshukor/data/recipe1m/text/vocab_all.txt
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       recipe_encoder: tfoodvicha-transformer
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       remove_additional_embedding: ['proj_image']
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       text_encoder: bert-base-uncased
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       vision_width: 768
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       vit_all_tokens: True
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       with_ingrs: True
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       with_instrs: True
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:       with_titles: True
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     with_classif: False
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:   optimizer:
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     batch_size_factor: None
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     clip_grad: 8.0
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     import: recipe1m.optimizers.factory
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     lr: 1e-05
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     lr_img: 1e-06
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     name: trijoint_fixed_fine_tune
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:     switch_epoch: 20
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:   path_opts: /home/mshukor/tfood/recipe1m/options/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_noimgproj.yaml
[S 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.75:   view: ['logs:train_epoch.loss', 'logs:train_epoch.bad_pairs', 'logs:eval_epoch.metric.med_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_1_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_5_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_10_im2recipe_mean', 'logs:eval_epoch.metric.med_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_1_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_5_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_10_recipe2im_mean', 'logs:optimizer.is_optimizer_recipe&image', 'logs:optimizer.total_norm']
[I 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.76: posix.uname_result(sysname='Linux', nodename='DGX', release='5.4.0-124-generic', version='#140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022', machine='x86_64')
[I 2022-10-28 13:30:06] ...ytorch/bootstrap/run.py.80: Available GPUs: [0]
[I 2022-10-28 13:30:06] ...strap/engines/factory.py.9: Creating engine...
[I 2022-10-28 13:30:06] ...trap/datasets/factory.py.8: Creating dataset...
[I 2022-10-28 13:30:07] ...m/datasets/recipe1m.py.152: Load VCs...
[I 2022-10-28 13:30:08] ...m/datasets/recipe1m.py.165: randkw_p... None
[I 2022-10-28 13:31:11] ...m/datasets/recipe1m.py.310: recipe elements to remove: [title]
[I 2022-10-28 13:31:11] ...rap/datasets/factory.py.21: Evaluation will take place on testset (51303 items)
[I 2022-10-28 13:31:11] ...strap/models/factory.py.21: Creating model...
[I 2022-10-28 13:31:12] ...e_networks/networks.py.203: Loading CLIP Visual model...
[I 2022-10-28 13:31:15] ...e_networks/networks.py.145: cuda
[I 2022-10-28 13:31:16] ...networks/tfood_vicha.py.49: Load VCs encoder...
[I 2022-10-28 13:31:19] ...els/metrics/trijoint.py.46: self.save_ids False
[I 2022-10-28 13:31:19] ...strap/models/factory.py.44: Enabling CUDA mode...
[I 2022-10-28 13:31:21] ...etworks/tfood_vicha.py.184: training aux kw modules
[I 2022-10-28 13:31:21] ...etworks/tfood_vicha.py.184: training aux kw modules
[I 2022-10-28 13:31:21] ...otstrap/views/factory.py.8: Creating views...
[I 2022-10-28 13:31:21] ...strap/engines/engine.py.98: Loading best_eval_epoch.metric.recall_at_1_im2recipe_mean checkpoint
[I 2022-10-28 13:31:21] ...trap/engines/engine.py.440: Loading model...
[I 2022-10-28 13:31:22] ...trap/engines/engine.py.453: Loading engine...
[I 2022-10-28 13:31:22] ...trap/engines/engine.py.108: Launching evaluation procedures
[I 2022-10-28 13:31:22] ...trap/engines/engine.py.268: Evaluating model on testset for epoch 116
[I 2022-10-28 13:31:22] ...m/datasets/recipe1m.py.400: Dataset will be sampled with "random" batch_sampler.
[I 2022-10-28 13:31:25] ...trap/engines/engine.py.321: eval: epoch 116 | batch 0/512
[I 2022-10-28 13:31:25] ...trap/engines/engine.py.325:       elapsed: 0:00:02 | left: 0:23:58
[I 2022-10-28 13:31:25] ...trap/engines/engine.py.326:       process: 2.80937 | load: 1.53376
[I 2022-10-28 13:31:28] ...trap/engines/engine.py.321: eval: epoch 116 | batch 10/512
[I 2022-10-28 13:31:28] ...trap/engines/engine.py.325:       elapsed: 0:00:06 | left: 0:04:57
[I 2022-10-28 13:31:28] ...trap/engines/engine.py.326:       process: 0.31937 | load: 0.00016
[I 2022-10-28 13:31:32] ...trap/engines/engine.py.321: eval: epoch 116 | batch 20/512
[I 2022-10-28 13:31:32] ...trap/engines/engine.py.325:       elapsed: 0:00:09 | left: 0:02:52
[I 2022-10-28 13:31:32] ...trap/engines/engine.py.326:       process: 0.31692 | load: 0.00016
[I 2022-10-28 13:31:35] ...trap/engines/engine.py.321: eval: epoch 116 | batch 30/512
[I 2022-10-28 13:31:35] ...trap/engines/engine.py.325:       elapsed: 0:00:12 | left: 0:02:37
[I 2022-10-28 13:31:35] ...trap/engines/engine.py.326:       process: 0.32107 | load: 0.00014
[I 2022-10-28 13:31:38] ...trap/engines/engine.py.321: eval: epoch 116 | batch 40/512
[I 2022-10-28 13:31:38] ...trap/engines/engine.py.325:       elapsed: 0:00:15 | left: 0:02:35
[I 2022-10-28 13:31:38] ...trap/engines/engine.py.326:       process: 0.31791 | load: 0.00014
[I 2022-10-28 13:31:41] ...trap/engines/engine.py.321: eval: epoch 116 | batch 50/512
[I 2022-10-28 13:31:41] ...trap/engines/engine.py.325:       elapsed: 0:00:19 | left: 0:02:29
[I 2022-10-28 13:31:41] ...trap/engines/engine.py.326:       process: 0.32508 | load: 0.00019
[I 2022-10-28 13:31:45] ...trap/engines/engine.py.321: eval: epoch 116 | batch 60/512
[I 2022-10-28 13:31:45] ...trap/engines/engine.py.325:       elapsed: 0:00:22 | left: 0:02:25
[I 2022-10-28 13:31:45] ...trap/engines/engine.py.326:       process: 0.31863 | load: 0.00014
[I 2022-10-28 13:31:48] ...trap/engines/engine.py.321: eval: epoch 116 | batch 70/512
[I 2022-10-28 13:31:48] ...trap/engines/engine.py.325:       elapsed: 0:00:25 | left: 0:02:27
[I 2022-10-28 13:31:48] ...trap/engines/engine.py.326:       process: 0.36837 | load: 0.00014
[I 2022-10-28 13:31:51] ...trap/engines/engine.py.321: eval: epoch 116 | batch 80/512
[I 2022-10-28 13:31:51] ...trap/engines/engine.py.325:       elapsed: 0:00:28 | left: 0:02:20
[I 2022-10-28 13:31:51] ...trap/engines/engine.py.326:       process: 0.32598 | load: 0.00017
[I 2022-10-28 13:31:55] ...trap/engines/engine.py.321: eval: epoch 116 | batch 90/512
[I 2022-10-28 13:31:55] ...trap/engines/engine.py.325:       elapsed: 0:00:32 | left: 0:02:18
[I 2022-10-28 13:31:55] ...trap/engines/engine.py.326:       process: 0.32999 | load: 0.00014
[I 2022-10-28 13:31:58] ...trap/engines/engine.py.321: eval: epoch 116 | batch 100/512
[I 2022-10-28 13:31:58] ...trap/engines/engine.py.325:       elapsed: 0:00:35 | left: 0:02:12
[I 2022-10-28 13:31:58] ...trap/engines/engine.py.326:       process: 0.28506 | load: 0.00017
[I 2022-10-28 13:32:01] ...trap/engines/engine.py.321: eval: epoch 116 | batch 110/512
[I 2022-10-28 13:32:01] ...trap/engines/engine.py.325:       elapsed: 0:00:38 | left: 0:01:54
[I 2022-10-28 13:32:01] ...trap/engines/engine.py.326:       process: 0.27902 | load: 0.00015
[I 2022-10-28 13:32:03] ...trap/engines/engine.py.321: eval: epoch 116 | batch 120/512
[I 2022-10-28 13:32:03] ...trap/engines/engine.py.325:       elapsed: 0:00:41 | left: 0:01:50
[I 2022-10-28 13:32:03] ...trap/engines/engine.py.326:       process: 0.28267 | load: 0.00015
[I 2022-10-28 13:32:06] ...trap/engines/engine.py.321: eval: epoch 116 | batch 130/512
[I 2022-10-28 13:32:06] ...trap/engines/engine.py.325:       elapsed: 0:00:43 | left: 0:01:47
[I 2022-10-28 13:32:06] ...trap/engines/engine.py.326:       process: 0.28282 | load: 0.00015
[I 2022-10-28 13:32:09] ...trap/engines/engine.py.321: eval: epoch 116 | batch 140/512
[I 2022-10-28 13:32:09] ...trap/engines/engine.py.325:       elapsed: 0:00:46 | left: 0:01:45
[I 2022-10-28 13:32:09] ...trap/engines/engine.py.326:       process: 0.27715 | load: 0.00015
[I 2022-10-28 13:32:12] ...trap/engines/engine.py.321: eval: epoch 116 | batch 150/512
[I 2022-10-28 13:32:12] ...trap/engines/engine.py.325:       elapsed: 0:00:49 | left: 0:01:42
[I 2022-10-28 13:32:12] ...trap/engines/engine.py.326:       process: 0.28258 | load: 0.00014
[I 2022-10-28 13:32:15] ...trap/engines/engine.py.321: eval: epoch 116 | batch 160/512
[I 2022-10-28 13:32:15] ...trap/engines/engine.py.325:       elapsed: 0:00:52 | left: 0:01:38
[I 2022-10-28 13:32:15] ...trap/engines/engine.py.326:       process: 0.27695 | load: 0.00016
[I 2022-10-28 13:32:18] ...trap/engines/engine.py.321: eval: epoch 116 | batch 170/512
[I 2022-10-28 13:32:18] ...trap/engines/engine.py.325:       elapsed: 0:00:55 | left: 0:01:35
[I 2022-10-28 13:32:18] ...trap/engines/engine.py.326:       process: 0.28085 | load: 0.00015
[I 2022-10-28 13:32:20] ...trap/engines/engine.py.321: eval: epoch 116 | batch 180/512
[I 2022-10-28 13:32:20] ...trap/engines/engine.py.325:       elapsed: 0:00:58 | left: 0:01:32
[I 2022-10-28 13:32:20] ...trap/engines/engine.py.326:       process: 0.27483 | load: 0.00016
[I 2022-10-28 13:32:23] ...trap/engines/engine.py.321: eval: epoch 116 | batch 190/512
[I 2022-10-28 13:32:23] ...trap/engines/engine.py.325:       elapsed: 0:01:00 | left: 0:01:30
[I 2022-10-28 13:32:23] ...trap/engines/engine.py.326:       process: 0.28270 | load: 0.00017
[I 2022-10-28 13:32:26] ...trap/engines/engine.py.321: eval: epoch 116 | batch 200/512
[I 2022-10-28 13:32:26] ...trap/engines/engine.py.325:       elapsed: 0:01:03 | left: 0:01:28
[I 2022-10-28 13:32:26] ...trap/engines/engine.py.326:       process: 0.28217 | load: 0.00019
[I 2022-10-28 13:32:29] ...trap/engines/engine.py.321: eval: epoch 116 | batch 210/512
[I 2022-10-28 13:32:29] ...trap/engines/engine.py.325:       elapsed: 0:01:06 | left: 0:01:25
[I 2022-10-28 13:32:29] ...trap/engines/engine.py.326:       process: 0.27945 | load: 0.00014
[I 2022-10-28 13:32:32] ...trap/engines/engine.py.321: eval: epoch 116 | batch 220/512
[I 2022-10-28 13:32:32] ...trap/engines/engine.py.325:       elapsed: 0:01:09 | left: 0:01:22
[I 2022-10-28 13:32:32] ...trap/engines/engine.py.326:       process: 0.28183 | load: 0.00017
[I 2022-10-28 13:32:34] ...trap/engines/engine.py.321: eval: epoch 116 | batch 230/512
[I 2022-10-28 13:32:34] ...trap/engines/engine.py.325:       elapsed: 0:01:12 | left: 0:01:19
[I 2022-10-28 13:32:34] ...trap/engines/engine.py.326:       process: 0.28392 | load: 0.00017
[I 2022-10-28 13:32:37] ...trap/engines/engine.py.321: eval: epoch 116 | batch 240/512
[I 2022-10-28 13:32:37] ...trap/engines/engine.py.325:       elapsed: 0:01:14 | left: 0:01:16
[I 2022-10-28 13:32:37] ...trap/engines/engine.py.326:       process: 0.28123 | load: 0.00015
[I 2022-10-28 13:32:40] ...trap/engines/engine.py.321: eval: epoch 116 | batch 250/512
[I 2022-10-28 13:32:40] ...trap/engines/engine.py.325:       elapsed: 0:01:17 | left: 0:01:13
[I 2022-10-28 13:32:40] ...trap/engines/engine.py.326:       process: 0.27778 | load: 0.00014
[I 2022-10-28 13:32:43] ...trap/engines/engine.py.321: eval: epoch 116 | batch 260/512
[I 2022-10-28 13:32:43] ...trap/engines/engine.py.325:       elapsed: 0:01:20 | left: 0:01:10
[I 2022-10-28 13:32:43] ...trap/engines/engine.py.326:       process: 0.27907 | load: 0.00015
[I 2022-10-28 13:32:46] ...trap/engines/engine.py.321: eval: epoch 116 | batch 270/512
[I 2022-10-28 13:32:46] ...trap/engines/engine.py.325:       elapsed: 0:01:23 | left: 0:01:07
[I 2022-10-28 13:32:46] ...trap/engines/engine.py.326:       process: 0.28220 | load: 0.00014
[I 2022-10-28 13:32:49] ...trap/engines/engine.py.321: eval: epoch 116 | batch 280/512
[I 2022-10-28 13:32:49] ...trap/engines/engine.py.325:       elapsed: 0:01:26 | left: 0:01:05
[I 2022-10-28 13:32:49] ...trap/engines/engine.py.326:       process: 0.27990 | load: 0.00017
[I 2022-10-28 13:32:51] ...trap/engines/engine.py.321: eval: epoch 116 | batch 290/512
[I 2022-10-28 13:32:51] ...trap/engines/engine.py.325:       elapsed: 0:01:29 | left: 0:01:02
[I 2022-10-28 13:32:51] ...trap/engines/engine.py.326:       process: 0.27802 | load: 0.00014
[I 2022-10-28 13:32:54] ...trap/engines/engine.py.321: eval: epoch 116 | batch 300/512
[I 2022-10-28 13:32:54] ...trap/engines/engine.py.325:       elapsed: 0:01:31 | left: 0:00:59
[I 2022-10-28 13:32:54] ...trap/engines/engine.py.326:       process: 0.27915 | load: 0.00018
[I 2022-10-28 13:32:57] ...trap/engines/engine.py.321: eval: epoch 116 | batch 310/512
[I 2022-10-28 13:32:57] ...trap/engines/engine.py.325:       elapsed: 0:01:34 | left: 0:00:56
[I 2022-10-28 13:32:57] ...trap/engines/engine.py.326:       process: 0.28357 | load: 0.00017
[I 2022-10-28 13:33:00] ...trap/engines/engine.py.321: eval: epoch 116 | batch 320/512
[I 2022-10-28 13:33:00] ...trap/engines/engine.py.325:       elapsed: 0:01:37 | left: 0:00:53
[I 2022-10-28 13:33:00] ...trap/engines/engine.py.326:       process: 0.27431 | load: 0.00016
[I 2022-10-28 13:33:03] ...trap/engines/engine.py.321: eval: epoch 116 | batch 330/512
[I 2022-10-28 13:33:03] ...trap/engines/engine.py.325:       elapsed: 0:01:40 | left: 0:00:50
[I 2022-10-28 13:33:03] ...trap/engines/engine.py.326:       process: 0.27851 | load: 0.00016
[I 2022-10-28 13:33:05] ...trap/engines/engine.py.321: eval: epoch 116 | batch 340/512
[I 2022-10-28 13:33:05] ...trap/engines/engine.py.325:       elapsed: 0:01:43 | left: 0:00:48
[I 2022-10-28 13:33:05] ...trap/engines/engine.py.326:       process: 0.28105 | load: 0.00017
[I 2022-10-28 13:33:08] ...trap/engines/engine.py.321: eval: epoch 116 | batch 350/512
[I 2022-10-28 13:33:08] ...trap/engines/engine.py.325:       elapsed: 0:01:45 | left: 0:00:45
[I 2022-10-28 13:33:08] ...trap/engines/engine.py.326:       process: 0.27911 | load: 0.00014
[I 2022-10-28 13:33:11] ...trap/engines/engine.py.321: eval: epoch 116 | batch 360/512
[I 2022-10-28 13:33:11] ...trap/engines/engine.py.325:       elapsed: 0:01:48 | left: 0:00:42
[I 2022-10-28 13:33:11] ...trap/engines/engine.py.326:       process: 0.27966 | load: 0.00016
[I 2022-10-28 13:33:14] ...trap/engines/engine.py.321: eval: epoch 116 | batch 370/512
[I 2022-10-28 13:33:14] ...trap/engines/engine.py.325:       elapsed: 0:01:51 | left: 0:00:39
[I 2022-10-28 13:33:14] ...trap/engines/engine.py.326:       process: 0.28560 | load: 0.00014
[I 2022-10-28 13:33:17] ...trap/engines/engine.py.321: eval: epoch 116 | batch 380/512
[I 2022-10-28 13:33:17] ...trap/engines/engine.py.325:       elapsed: 0:01:54 | left: 0:00:37
[I 2022-10-28 13:33:17] ...trap/engines/engine.py.326:       process: 0.27885 | load: 0.00017
[I 2022-10-28 13:33:19] ...trap/engines/engine.py.321: eval: epoch 116 | batch 390/512
[I 2022-10-28 13:33:19] ...trap/engines/engine.py.325:       elapsed: 0:01:57 | left: 0:00:34
[I 2022-10-28 13:33:19] ...trap/engines/engine.py.326:       process: 0.28018 | load: 0.00014
[I 2022-10-28 13:33:22] ...trap/engines/engine.py.321: eval: epoch 116 | batch 400/512
[I 2022-10-28 13:33:22] ...trap/engines/engine.py.325:       elapsed: 0:01:59 | left: 0:00:31
[I 2022-10-28 13:33:22] ...trap/engines/engine.py.326:       process: 0.27432 | load: 0.00016
[I 2022-10-28 13:33:25] ...trap/engines/engine.py.321: eval: epoch 116 | batch 410/512
[I 2022-10-28 13:33:25] ...trap/engines/engine.py.325:       elapsed: 0:02:02 | left: 0:00:28
[I 2022-10-28 13:33:25] ...trap/engines/engine.py.326:       process: 0.28391 | load: 0.00015
[I 2022-10-28 13:33:28] ...trap/engines/engine.py.321: eval: epoch 116 | batch 420/512
[I 2022-10-28 13:33:28] ...trap/engines/engine.py.325:       elapsed: 0:02:05 | left: 0:00:25
[I 2022-10-28 13:33:28] ...trap/engines/engine.py.326:       process: 0.27875 | load: 0.00017
[I 2022-10-28 13:33:31] ...trap/engines/engine.py.321: eval: epoch 116 | batch 430/512
[I 2022-10-28 13:33:31] ...trap/engines/engine.py.325:       elapsed: 0:02:08 | left: 0:00:23
[I 2022-10-28 13:33:31] ...trap/engines/engine.py.326:       process: 0.28249 | load: 0.00016
[I 2022-10-28 13:33:34] ...trap/engines/engine.py.321: eval: epoch 116 | batch 440/512
[I 2022-10-28 13:33:34] ...trap/engines/engine.py.325:       elapsed: 0:02:11 | left: 0:00:20
[I 2022-10-28 13:33:34] ...trap/engines/engine.py.326:       process: 0.28002 | load: 0.00016
[I 2022-10-28 13:33:36] ...trap/engines/engine.py.321: eval: epoch 116 | batch 450/512
[I 2022-10-28 13:33:36] ...trap/engines/engine.py.325:       elapsed: 0:02:13 | left: 0:00:17
[I 2022-10-28 13:33:36] ...trap/engines/engine.py.326:       process: 0.27604 | load: 0.00016
[I 2022-10-28 13:33:39] ...trap/engines/engine.py.321: eval: epoch 116 | batch 460/512
[I 2022-10-28 13:33:39] ...trap/engines/engine.py.325:       elapsed: 0:02:16 | left: 0:00:14
[I 2022-10-28 13:33:39] ...trap/engines/engine.py.326:       process: 0.27892 | load: 0.00017
[I 2022-10-28 13:33:42] ...trap/engines/engine.py.321: eval: epoch 116 | batch 470/512
[I 2022-10-28 13:33:42] ...trap/engines/engine.py.325:       elapsed: 0:02:19 | left: 0:00:11
[I 2022-10-28 13:33:42] ...trap/engines/engine.py.326:       process: 0.27815 | load: 0.00015
[I 2022-10-28 13:33:45] ...trap/engines/engine.py.321: eval: epoch 116 | batch 480/512
[I 2022-10-28 13:33:45] ...trap/engines/engine.py.325:       elapsed: 0:02:22 | left: 0:00:08
[I 2022-10-28 13:33:45] ...trap/engines/engine.py.326:       process: 0.27961 | load: 0.00017
[I 2022-10-28 13:33:48] ...trap/engines/engine.py.321: eval: epoch 116 | batch 490/512
[I 2022-10-28 13:33:48] ...trap/engines/engine.py.325:       elapsed: 0:02:25 | left: 0:00:06
[I 2022-10-28 13:33:48] ...trap/engines/engine.py.326:       process: 0.27616 | load: 0.00018
[I 2022-10-28 13:33:50] ...trap/engines/engine.py.321: eval: epoch 116 | batch 500/512
[I 2022-10-28 13:33:50] ...trap/engines/engine.py.325:       elapsed: 0:02:27 | left: 0:00:03
[I 2022-10-28 13:33:50] ...trap/engines/engine.py.326:       process: 0.27706 | load: 0.00018
[I 2022-10-28 13:33:53] ...trap/engines/engine.py.321: eval: epoch 116 | batch 510/512
[I 2022-10-28 13:33:53] ...trap/engines/engine.py.325:       elapsed: 0:02:30 | left: 0:00:00
[I 2022-10-28 13:33:53] ...trap/engines/engine.py.326:       process: 0.27323 | load: 0.00010
[S 2022-10-28 13:33:54] ...trap/engines/engine.py.113: eval_epoch.epoch: 116
[I 2022-10-28 13:33:54] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs
[S 2022-10-28 13:33:58] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00
[S 2022-10-28 13:33:58] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00
[S 2022-10-28 13:33:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.684100
[S 2022-10-28 13:33:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.876000
[S 2022-10-28 13:33:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.912600
[S 2022-10-28 13:33:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.686400
[S 2022-10-28 13:33:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.880500
[S 2022-10-28 13:33:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.919700
[S 2022-10-28 13:33:58] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000
[S 2022-10-28 13:33:58] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000
[S 2022-10-28 13:33:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.017207
[S 2022-10-28 13:33:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007616
[S 2022-10-28 13:33:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.005713
[S 2022-10-28 13:33:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.012068
[S 2022-10-28 13:33:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.010837
[S 2022-10-28 13:33:58] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.005100
[I 2022-10-28 13:33:59] ...trap/engines/engine.py.115: Ending evaluation procedures
[W 2022-10-28 13:33:59] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json'
[W 2022-10-28 13:33:59] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[W 2022-10-28 13:33:59] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[I 2022-10-28 13:34:03] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html
[I 2022-10-28 13:34:47] ...ytorch/bootstrap/run.py.73: Saving environment info
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75: options: 
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:   dataset:
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     aux_kw_path:
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       test: /home/mshukor/data/recipe1m/clip_da/layer1_test_titles_kw.json
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       train: /home/mshukor/data/recipe1m/clip_da/layer1_train_titles_kw.json
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       val: /home/mshukor/data/recipe1m/clip_da/layer1_val_titles_kw.json
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     batch_sampler: triplet_classif
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     batch_size: 100
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     database: lmdb
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     dir: /home/mshukor/data/recipe1m
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     eval_split: test
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     freq_mismatch: 0.0
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     image_from: database
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     import: recipe1m.datasets.factory
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     interchange_ingrd_instr: None
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     kw_path:
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       test: /home/mshukor/data/recipe1m/clip_da/layer1_test_ingr_kw.json
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       train: /home/mshukor/data/recipe1m/clip_da/layer1_train_ingr_kw.json
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       val: /home/mshukor/data/recipe1m/clip_da/layer1_val_ingr_kw.json
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     name: recipe1m
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     nb_classes: 1048
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     nb_threads: 4
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     randkw_p: 0.3
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     randkw_p_aux: 0.5
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     remove_list: [ingredients]
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     tokenized_raw_text: True
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     train_split: None
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     use_vcs: True
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:   engine:
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     name: logger
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     nb_epochs: 120
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     print_freq: 10
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     saving_criteria: ['eval_epoch.metric.med_im2recipe_mean:min', 'eval_epoch.metric.recall_at_1_im2recipe_mean:max']
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:   exp:
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     checkpoint: None
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     dir: /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     resume: best_eval_epoch.metric.recall_at_1_im2recipe_mean
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:   misc:
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     cuda: True
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     data_parrallel: True
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     logs_name: None
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     overrite: False
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     seed: 1338
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:   model:
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     criterion:
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:         tri_cross_encoder: False
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       itm_loss_weight: 1
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       keep_background: False
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       retrieval_strategy:
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:         aggregation: valid
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:         get_tokens: True
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:         margin: 0.05
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:         margin_params:
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:           increment: 0.005
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:           increment_margin: True
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:           max_margin: 0.3
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:         name: triplet
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:         nb_samples: 9999
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:         sampling: max_negative
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:         substrategy: ['IRR', 'RII', 'SIRR', 'SRII']
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:         substrategy_weights: [1.0, 1.0, 0.1, 0.1]
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     import: recipe1m.models.factory
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     metric:
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       k_test: 10
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       nb_bags: 10
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       nb_matchs_per_bag: 1000
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       retrieval_dir: /tmp/recipe1m
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       save_ids: False
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       trijoint: True
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     name: trijoint
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     network:
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       activations: ['tanh', 'normalize']
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       aux_kwords: True
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       aux_kwords_encoder: True
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       avg_concat: True
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       bert_config: /home/mshukor/tfood/recipe1m/models/networks/recipe_networks/config_bert_albef.json
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       cat_pos: 0
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       checkpoint: None
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       class_attention: False
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       class_attention_fus: False
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       cls_token: False
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       cls_token_fus: False
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       cross_decoder_image: True
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       cross_decoder_recipe: True
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:         class_attention: False
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:         cls_token: False
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:         cross_attention: False
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:         cross_decoder: True
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:         double_decoder: False
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:         get_tokens: False
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:         load_backbones: False
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:         n_heads: 4
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:         n_layers: 4
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:         query_img: False
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       cross_transformer: False
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       dim_emb: 1024
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       dim_image_out: 2048
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       dim_ingr_out: 300
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       dim_instr_in: 1024
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       dim_instr_out: 1024
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       freeze_im: False
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       freeze_rec: False
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       get_tokens: True
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       get_tokens_cross_decoder_recipe: True
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       hidden_size: 512
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       image_backbone_name: clip
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       image_res: 224
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       kwords_same_level: False
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       last_cat_aux: True
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       n_heads: 4
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       n_heads_cross: 4
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       n_heads_cross_image: 4
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       n_heads_single: 4
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       n_layers: 2
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       n_layers_cross: 2
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       n_layers_cross_image: 1
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       n_layers_single: 2
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       num_hidden_layers_kw: 2
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       old: True
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       path_ingrs: /home/mshukor/data/recipe1m/text/vocab.pkl
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       path_vocab: /home/mshukor/data/recipe1m/text/vocab_all.txt
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       recipe_encoder: tfoodvicha-transformer
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       remove_additional_embedding: ['proj_image']
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       text_encoder: bert-base-uncased
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       vision_width: 768
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       vit_all_tokens: True
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       with_ingrs: True
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       with_instrs: True
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:       with_titles: True
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     with_classif: False
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:   optimizer:
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     batch_size_factor: None
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     clip_grad: 8.0
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     import: recipe1m.optimizers.factory
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     lr: 1e-05
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     lr_img: 1e-06
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     name: trijoint_fixed_fine_tune
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:     switch_epoch: 20
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:   path_opts: /home/mshukor/tfood/recipe1m/options/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_noimgproj.yaml
[S 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.75:   view: ['logs:train_epoch.loss', 'logs:train_epoch.bad_pairs', 'logs:eval_epoch.metric.med_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_1_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_5_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_10_im2recipe_mean', 'logs:eval_epoch.metric.med_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_1_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_5_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_10_recipe2im_mean', 'logs:optimizer.is_optimizer_recipe&image', 'logs:optimizer.total_norm']
[I 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.76: posix.uname_result(sysname='Linux', nodename='DGX', release='5.4.0-124-generic', version='#140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022', machine='x86_64')
[I 2022-10-28 13:34:48] ...ytorch/bootstrap/run.py.80: Available GPUs: [0]
[I 2022-10-28 13:34:48] ...strap/engines/factory.py.9: Creating engine...
[I 2022-10-28 13:34:48] ...trap/datasets/factory.py.8: Creating dataset...
[I 2022-10-28 13:34:50] ...m/datasets/recipe1m.py.152: Load VCs...
[I 2022-10-28 13:34:50] ...m/datasets/recipe1m.py.165: randkw_p... None
[I 2022-10-28 13:35:53] ...m/datasets/recipe1m.py.310: recipe elements to remove: [ingredients]
[I 2022-10-28 13:35:53] ...rap/datasets/factory.py.21: Evaluation will take place on testset (51303 items)
[I 2022-10-28 13:35:53] ...strap/models/factory.py.21: Creating model...
[I 2022-10-28 13:35:54] ...e_networks/networks.py.203: Loading CLIP Visual model...
[I 2022-10-28 13:35:57] ...e_networks/networks.py.145: cuda
[I 2022-10-28 13:35:59] ...networks/tfood_vicha.py.49: Load VCs encoder...
[I 2022-10-28 13:36:01] ...els/metrics/trijoint.py.46: self.save_ids False
[I 2022-10-28 13:36:01] ...strap/models/factory.py.44: Enabling CUDA mode...
[I 2022-10-28 13:36:03] ...etworks/tfood_vicha.py.184: training aux kw modules
[I 2022-10-28 13:36:03] ...etworks/tfood_vicha.py.184: training aux kw modules
[I 2022-10-28 13:36:03] ...otstrap/views/factory.py.8: Creating views...
[I 2022-10-28 13:36:04] ...strap/engines/engine.py.98: Loading best_eval_epoch.metric.recall_at_1_im2recipe_mean checkpoint
[I 2022-10-28 13:36:04] ...trap/engines/engine.py.440: Loading model...
[I 2022-10-28 13:36:05] ...trap/engines/engine.py.453: Loading engine...
[I 2022-10-28 13:36:05] ...trap/engines/engine.py.108: Launching evaluation procedures
[I 2022-10-28 13:36:05] ...trap/engines/engine.py.268: Evaluating model on testset for epoch 116
[I 2022-10-28 13:36:05] ...m/datasets/recipe1m.py.400: Dataset will be sampled with "random" batch_sampler.
[I 2022-10-28 13:36:07] ...trap/engines/engine.py.321: eval: epoch 116 | batch 0/512
[I 2022-10-28 13:36:07] ...trap/engines/engine.py.325:       elapsed: 0:00:02 | left: 0:23:18
[I 2022-10-28 13:36:07] ...trap/engines/engine.py.326:       process: 2.73153 | load: 1.47562
[I 2022-10-28 13:36:10] ...trap/engines/engine.py.321: eval: epoch 116 | batch 10/512
[I 2022-10-28 13:36:10] ...trap/engines/engine.py.325:       elapsed: 0:00:05 | left: 0:04:42
[I 2022-10-28 13:36:10] ...trap/engines/engine.py.326:       process: 0.30074 | load: 0.00017
[I 2022-10-28 13:36:13] ...trap/engines/engine.py.321: eval: epoch 116 | batch 20/512
[I 2022-10-28 13:36:13] ...trap/engines/engine.py.325:       elapsed: 0:00:08 | left: 0:02:41
[I 2022-10-28 13:36:13] ...trap/engines/engine.py.326:       process: 0.29273 | load: 0.00014
[I 2022-10-28 13:36:16] ...trap/engines/engine.py.321: eval: epoch 116 | batch 30/512
[I 2022-10-28 13:36:16] ...trap/engines/engine.py.325:       elapsed: 0:00:11 | left: 0:02:28
[I 2022-10-28 13:36:16] ...trap/engines/engine.py.326:       process: 0.29684 | load: 0.00016
[I 2022-10-28 13:36:19] ...trap/engines/engine.py.321: eval: epoch 116 | batch 40/512
[I 2022-10-28 13:36:19] ...trap/engines/engine.py.325:       elapsed: 0:00:14 | left: 0:02:22
[I 2022-10-28 13:36:19] ...trap/engines/engine.py.326:       process: 0.29948 | load: 0.00015
[I 2022-10-28 13:36:22] ...trap/engines/engine.py.321: eval: epoch 116 | batch 50/512
[I 2022-10-28 13:36:22] ...trap/engines/engine.py.325:       elapsed: 0:00:17 | left: 0:02:19
[I 2022-10-28 13:36:22] ...trap/engines/engine.py.326:       process: 0.29775 | load: 0.00018
[I 2022-10-28 13:36:25] ...trap/engines/engine.py.321: eval: epoch 116 | batch 60/512
[I 2022-10-28 13:36:25] ...trap/engines/engine.py.325:       elapsed: 0:00:20 | left: 0:02:16
[I 2022-10-28 13:36:25] ...trap/engines/engine.py.326:       process: 0.30389 | load: 0.00016
[I 2022-10-28 13:36:28] ...trap/engines/engine.py.321: eval: epoch 116 | batch 70/512
[I 2022-10-28 13:36:28] ...trap/engines/engine.py.325:       elapsed: 0:00:23 | left: 0:02:14
[I 2022-10-28 13:36:28] ...trap/engines/engine.py.326:       process: 0.29899 | load: 0.00015
[I 2022-10-28 13:36:32] ...trap/engines/engine.py.321: eval: epoch 116 | batch 80/512
[I 2022-10-28 13:36:32] ...trap/engines/engine.py.325:       elapsed: 0:00:27 | left: 0:02:14
[I 2022-10-28 13:36:32] ...trap/engines/engine.py.326:       process: 0.34136 | load: 0.00014
[I 2022-10-28 13:36:35] ...trap/engines/engine.py.321: eval: epoch 116 | batch 90/512
[I 2022-10-28 13:36:35] ...trap/engines/engine.py.325:       elapsed: 0:00:30 | left: 0:02:07
[I 2022-10-28 13:36:35] ...trap/engines/engine.py.326:       process: 0.30357 | load: 0.00017
[I 2022-10-28 13:36:38] ...trap/engines/engine.py.321: eval: epoch 116 | batch 100/512
[I 2022-10-28 13:36:38] ...trap/engines/engine.py.325:       elapsed: 0:00:33 | left: 0:02:01
[I 2022-10-28 13:36:38] ...trap/engines/engine.py.326:       process: 0.25887 | load: 0.00015
[I 2022-10-28 13:36:40] ...trap/engines/engine.py.321: eval: epoch 116 | batch 110/512
[I 2022-10-28 13:36:40] ...trap/engines/engine.py.325:       elapsed: 0:00:35 | left: 0:01:46
[I 2022-10-28 13:36:40] ...trap/engines/engine.py.326:       process: 0.26453 | load: 0.00014
[I 2022-10-28 13:36:43] ...trap/engines/engine.py.321: eval: epoch 116 | batch 120/512
[I 2022-10-28 13:36:43] ...trap/engines/engine.py.325:       elapsed: 0:00:38 | left: 0:01:42
[I 2022-10-28 13:36:43] ...trap/engines/engine.py.326:       process: 0.25894 | load: 0.00014
[I 2022-10-28 13:36:45] ...trap/engines/engine.py.321: eval: epoch 116 | batch 130/512
[I 2022-10-28 13:36:45] ...trap/engines/engine.py.325:       elapsed: 0:00:40 | left: 0:01:40
[I 2022-10-28 13:36:45] ...trap/engines/engine.py.326:       process: 0.27880 | load: 0.00014
[I 2022-10-28 13:36:48] ...trap/engines/engine.py.321: eval: epoch 116 | batch 140/512
[I 2022-10-28 13:36:48] ...trap/engines/engine.py.325:       elapsed: 0:00:43 | left: 0:01:37
[I 2022-10-28 13:36:48] ...trap/engines/engine.py.326:       process: 0.25561 | load: 0.00017
[I 2022-10-28 13:36:51] ...trap/engines/engine.py.321: eval: epoch 116 | batch 150/512
[I 2022-10-28 13:36:51] ...trap/engines/engine.py.325:       elapsed: 0:00:46 | left: 0:01:35
[I 2022-10-28 13:36:51] ...trap/engines/engine.py.326:       process: 0.26159 | load: 0.00022
[I 2022-10-28 13:36:53] ...trap/engines/engine.py.321: eval: epoch 116 | batch 160/512
[I 2022-10-28 13:36:53] ...trap/engines/engine.py.325:       elapsed: 0:00:48 | left: 0:01:36
[I 2022-10-28 13:36:53] ...trap/engines/engine.py.326:       process: 0.27285 | load: 0.00020
[I 2022-10-28 13:36:56] ...trap/engines/engine.py.321: eval: epoch 116 | batch 170/512
[I 2022-10-28 13:36:56] ...trap/engines/engine.py.325:       elapsed: 0:00:51 | left: 0:01:29
[I 2022-10-28 13:36:56] ...trap/engines/engine.py.326:       process: 0.26262 | load: 0.00014
[I 2022-10-28 13:36:59] ...trap/engines/engine.py.321: eval: epoch 116 | batch 180/512
[I 2022-10-28 13:36:59] ...trap/engines/engine.py.325:       elapsed: 0:00:54 | left: 0:01:26
[I 2022-10-28 13:36:59] ...trap/engines/engine.py.326:       process: 0.25698 | load: 0.00015
[I 2022-10-28 13:37:01] ...trap/engines/engine.py.321: eval: epoch 116 | batch 190/512
[I 2022-10-28 13:37:01] ...trap/engines/engine.py.325:       elapsed: 0:00:56 | left: 0:01:24
[I 2022-10-28 13:37:01] ...trap/engines/engine.py.326:       process: 0.26202 | load: 0.00017
[I 2022-10-28 13:37:04] ...trap/engines/engine.py.321: eval: epoch 116 | batch 200/512
[I 2022-10-28 13:37:04] ...trap/engines/engine.py.325:       elapsed: 0:00:59 | left: 0:01:26
[I 2022-10-28 13:37:04] ...trap/engines/engine.py.326:       process: 0.33431 | load: 0.00017
[I 2022-10-28 13:37:07] ...trap/engines/engine.py.321: eval: epoch 116 | batch 210/512
[I 2022-10-28 13:37:07] ...trap/engines/engine.py.325:       elapsed: 0:01:01 | left: 0:01:19
[I 2022-10-28 13:37:07] ...trap/engines/engine.py.326:       process: 0.26391 | load: 0.00017
[I 2022-10-28 13:37:09] ...trap/engines/engine.py.321: eval: epoch 116 | batch 220/512
[I 2022-10-28 13:37:09] ...trap/engines/engine.py.325:       elapsed: 0:01:04 | left: 0:01:16
[I 2022-10-28 13:37:09] ...trap/engines/engine.py.326:       process: 0.25771 | load: 0.00017
[I 2022-10-28 13:37:12] ...trap/engines/engine.py.321: eval: epoch 116 | batch 230/512
[I 2022-10-28 13:37:12] ...trap/engines/engine.py.325:       elapsed: 0:01:07 | left: 0:01:13
[I 2022-10-28 13:37:12] ...trap/engines/engine.py.326:       process: 0.26698 | load: 0.00014
[I 2022-10-28 13:37:14] ...trap/engines/engine.py.321: eval: epoch 116 | batch 240/512
[I 2022-10-28 13:37:14] ...trap/engines/engine.py.325:       elapsed: 0:01:09 | left: 0:01:10
[I 2022-10-28 13:37:14] ...trap/engines/engine.py.326:       process: 0.25734 | load: 0.00014
[I 2022-10-28 13:37:17] ...trap/engines/engine.py.321: eval: epoch 116 | batch 250/512
[I 2022-10-28 13:37:17] ...trap/engines/engine.py.325:       elapsed: 0:01:12 | left: 0:01:09
[I 2022-10-28 13:37:17] ...trap/engines/engine.py.326:       process: 0.28117 | load: 0.00014
[I 2022-10-28 13:37:20] ...trap/engines/engine.py.321: eval: epoch 116 | batch 260/512
[I 2022-10-28 13:37:20] ...trap/engines/engine.py.325:       elapsed: 0:01:15 | left: 0:01:05
[I 2022-10-28 13:37:20] ...trap/engines/engine.py.326:       process: 0.25727 | load: 0.00015
[I 2022-10-28 13:37:22] ...trap/engines/engine.py.321: eval: epoch 116 | batch 270/512
[I 2022-10-28 13:37:22] ...trap/engines/engine.py.325:       elapsed: 0:01:17 | left: 0:01:03
[I 2022-10-28 13:37:22] ...trap/engines/engine.py.326:       process: 0.27063 | load: 0.00015
[I 2022-10-28 13:37:25] ...trap/engines/engine.py.321: eval: epoch 116 | batch 280/512
[I 2022-10-28 13:37:25] ...trap/engines/engine.py.325:       elapsed: 0:01:20 | left: 0:01:00
[I 2022-10-28 13:37:25] ...trap/engines/engine.py.326:       process: 0.25567 | load: 0.00014
[I 2022-10-28 13:37:27] ...trap/engines/engine.py.321: eval: epoch 116 | batch 290/512
[I 2022-10-28 13:37:27] ...trap/engines/engine.py.325:       elapsed: 0:01:22 | left: 0:00:57
[I 2022-10-28 13:37:27] ...trap/engines/engine.py.326:       process: 0.26247 | load: 0.00016
[I 2022-10-28 13:37:30] ...trap/engines/engine.py.321: eval: epoch 116 | batch 300/512
[I 2022-10-28 13:37:30] ...trap/engines/engine.py.325:       elapsed: 0:01:25 | left: 0:00:55
[I 2022-10-28 13:37:30] ...trap/engines/engine.py.326:       process: 0.25422 | load: 0.00016
[I 2022-10-28 13:37:33] ...trap/engines/engine.py.321: eval: epoch 116 | batch 310/512
[I 2022-10-28 13:37:33] ...trap/engines/engine.py.325:       elapsed: 0:01:28 | left: 0:00:52
[I 2022-10-28 13:37:33] ...trap/engines/engine.py.326:       process: 0.25982 | load: 0.00017
[I 2022-10-28 13:37:35] ...trap/engines/engine.py.321: eval: epoch 116 | batch 320/512
[I 2022-10-28 13:37:35] ...trap/engines/engine.py.325:       elapsed: 0:01:30 | left: 0:00:49
[I 2022-10-28 13:37:35] ...trap/engines/engine.py.326:       process: 0.25729 | load: 0.00015
[I 2022-10-28 13:37:38] ...trap/engines/engine.py.321: eval: epoch 116 | batch 330/512
[I 2022-10-28 13:37:38] ...trap/engines/engine.py.325:       elapsed: 0:01:33 | left: 0:00:47
[I 2022-10-28 13:37:38] ...trap/engines/engine.py.326:       process: 0.25812 | load: 0.00017
[I 2022-10-28 13:37:40] ...trap/engines/engine.py.321: eval: epoch 116 | batch 340/512
[I 2022-10-28 13:37:40] ...trap/engines/engine.py.325:       elapsed: 0:01:35 | left: 0:00:47
[I 2022-10-28 13:37:40] ...trap/engines/engine.py.326:       process: 0.34833 | load: 0.00017
[I 2022-10-28 13:37:43] ...trap/engines/engine.py.321: eval: epoch 116 | batch 350/512
[I 2022-10-28 13:37:43] ...trap/engines/engine.py.325:       elapsed: 0:01:38 | left: 0:00:42
[I 2022-10-28 13:37:43] ...trap/engines/engine.py.326:       process: 0.26304 | load: 0.00016
[I 2022-10-28 13:37:46] ...trap/engines/engine.py.321: eval: epoch 116 | batch 360/512
[I 2022-10-28 13:37:46] ...trap/engines/engine.py.325:       elapsed: 0:01:41 | left: 0:00:39
[I 2022-10-28 13:37:46] ...trap/engines/engine.py.326:       process: 0.26014 | load: 0.00016
[I 2022-10-28 13:37:48] ...trap/engines/engine.py.321: eval: epoch 116 | batch 370/512
[I 2022-10-28 13:37:48] ...trap/engines/engine.py.325:       elapsed: 0:01:43 | left: 0:00:36
[I 2022-10-28 13:37:48] ...trap/engines/engine.py.326:       process: 0.26136 | load: 0.00015
[I 2022-10-28 13:37:51] ...trap/engines/engine.py.321: eval: epoch 116 | batch 380/512
[I 2022-10-28 13:37:51] ...trap/engines/engine.py.325:       elapsed: 0:01:46 | left: 0:00:34
[I 2022-10-28 13:37:51] ...trap/engines/engine.py.326:       process: 0.25895 | load: 0.00016
[I 2022-10-28 13:37:54] ...trap/engines/engine.py.321: eval: epoch 116 | batch 390/512
[I 2022-10-28 13:37:54] ...trap/engines/engine.py.325:       elapsed: 0:01:49 | left: 0:00:31
[I 2022-10-28 13:37:54] ...trap/engines/engine.py.326:       process: 0.26343 | load: 0.00015
[I 2022-10-28 13:37:56] ...trap/engines/engine.py.321: eval: epoch 116 | batch 400/512
[I 2022-10-28 13:37:56] ...trap/engines/engine.py.325:       elapsed: 0:01:51 | left: 0:00:29
[I 2022-10-28 13:37:56] ...trap/engines/engine.py.326:       process: 0.25883 | load: 0.00015
[I 2022-10-28 13:37:59] ...trap/engines/engine.py.321: eval: epoch 116 | batch 410/512
[I 2022-10-28 13:37:59] ...trap/engines/engine.py.325:       elapsed: 0:01:54 | left: 0:00:26
[I 2022-10-28 13:37:59] ...trap/engines/engine.py.326:       process: 0.26130 | load: 0.00018
[I 2022-10-28 13:38:01] ...trap/engines/engine.py.321: eval: epoch 116 | batch 420/512
[I 2022-10-28 13:38:01] ...trap/engines/engine.py.325:       elapsed: 0:01:56 | left: 0:00:23
[I 2022-10-28 13:38:01] ...trap/engines/engine.py.326:       process: 0.25655 | load: 0.00018
[I 2022-10-28 13:38:04] ...trap/engines/engine.py.321: eval: epoch 116 | batch 430/512
[I 2022-10-28 13:38:04] ...trap/engines/engine.py.325:       elapsed: 0:01:59 | left: 0:00:21
[I 2022-10-28 13:38:04] ...trap/engines/engine.py.326:       process: 0.26035 | load: 0.00017
[I 2022-10-28 13:38:07] ...trap/engines/engine.py.321: eval: epoch 116 | batch 440/512
[I 2022-10-28 13:38:07] ...trap/engines/engine.py.325:       elapsed: 0:02:02 | left: 0:00:18
[I 2022-10-28 13:38:07] ...trap/engines/engine.py.326:       process: 0.26088 | load: 0.00018
[I 2022-10-28 13:38:09] ...trap/engines/engine.py.321: eval: epoch 116 | batch 450/512
[I 2022-10-28 13:38:09] ...trap/engines/engine.py.325:       elapsed: 0:02:04 | left: 0:00:16
[I 2022-10-28 13:38:09] ...trap/engines/engine.py.326:       process: 0.25955 | load: 0.00015
[I 2022-10-28 13:38:12] ...trap/engines/engine.py.321: eval: epoch 116 | batch 460/512
[I 2022-10-28 13:38:12] ...trap/engines/engine.py.325:       elapsed: 0:02:07 | left: 0:00:13
[I 2022-10-28 13:38:12] ...trap/engines/engine.py.326:       process: 0.25786 | load: 0.00016
[I 2022-10-28 13:38:14] ...trap/engines/engine.py.321: eval: epoch 116 | batch 470/512
[I 2022-10-28 13:38:14] ...trap/engines/engine.py.325:       elapsed: 0:02:09 | left: 0:00:10
[I 2022-10-28 13:38:14] ...trap/engines/engine.py.326:       process: 0.26250 | load: 0.00019
[I 2022-10-28 13:38:17] ...trap/engines/engine.py.321: eval: epoch 116 | batch 480/512
[I 2022-10-28 13:38:17] ...trap/engines/engine.py.325:       elapsed: 0:02:12 | left: 0:00:08
[I 2022-10-28 13:38:17] ...trap/engines/engine.py.326:       process: 0.26159 | load: 0.00017
[I 2022-10-28 13:38:20] ...trap/engines/engine.py.321: eval: epoch 116 | batch 490/512
[I 2022-10-28 13:38:20] ...trap/engines/engine.py.325:       elapsed: 0:02:15 | left: 0:00:06
[I 2022-10-28 13:38:20] ...trap/engines/engine.py.326:       process: 0.37571 | load: 0.00016
[I 2022-10-28 13:38:22] ...trap/engines/engine.py.321: eval: epoch 116 | batch 500/512
[I 2022-10-28 13:38:22] ...trap/engines/engine.py.325:       elapsed: 0:02:17 | left: 0:00:03
[I 2022-10-28 13:38:22] ...trap/engines/engine.py.326:       process: 0.25807 | load: 0.00014
[I 2022-10-28 13:38:25] ...trap/engines/engine.py.321: eval: epoch 116 | batch 510/512
[I 2022-10-28 13:38:25] ...trap/engines/engine.py.325:       elapsed: 0:02:20 | left: 0:00:00
[I 2022-10-28 13:38:25] ...trap/engines/engine.py.326:       process: 0.25905 | load: 0.00010
[S 2022-10-28 13:38:26] ...trap/engines/engine.py.113: eval_epoch.epoch: 116
[I 2022-10-28 13:38:26] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs
[S 2022-10-28 13:38:30] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 4.80
[S 2022-10-28 13:38:30] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 2.40
[S 2022-10-28 13:38:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.249300
[S 2022-10-28 13:38:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.524300
[S 2022-10-28 13:38:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.639600
[S 2022-10-28 13:38:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.364800
[S 2022-10-28 13:38:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.657300
[S 2022-10-28 13:38:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.753300
[S 2022-10-28 13:38:30] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.400000
[S 2022-10-28 13:38:30] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.489898
[S 2022-10-28 13:38:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.012554
[S 2022-10-28 13:38:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.013682
[S 2022-10-28 13:38:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.010356
[S 2022-10-28 13:38:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.012311
[S 2022-10-28 13:38:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.012458
[S 2022-10-28 13:38:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.012673
[I 2022-10-28 13:38:30] ...trap/engines/engine.py.115: Ending evaluation procedures
[W 2022-10-28 13:38:31] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json'
[W 2022-10-28 13:38:31] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[W 2022-10-28 13:38:31] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[I 2022-10-28 13:38:35] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html
[I 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.73: Saving environment info
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75: options: 
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:   dataset:
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     aux_kw_path:
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       test: /home/mshukor/data/recipe1m/clip_da/layer1_test_titles_kw.json
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       train: /home/mshukor/data/recipe1m/clip_da/layer1_train_titles_kw.json
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       val: /home/mshukor/data/recipe1m/clip_da/layer1_val_titles_kw.json
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     batch_sampler: triplet_classif
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     batch_size: 100
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     database: lmdb
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     dir: /home/mshukor/data/recipe1m
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     eval_split: test
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     freq_mismatch: 0.0
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     image_from: database
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     import: recipe1m.datasets.factory
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     interchange_ingrd_instr: None
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     kw_path:
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       test: /home/mshukor/data/recipe1m/clip_da/layer1_test_ingr_kw.json
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       train: /home/mshukor/data/recipe1m/clip_da/layer1_train_ingr_kw.json
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       val: /home/mshukor/data/recipe1m/clip_da/layer1_val_ingr_kw.json
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     name: recipe1m
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     nb_classes: 1048
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     nb_threads: 4
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     randkw_p: 0.3
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     randkw_p_aux: 0.5
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     remove_list: [title,instructions]
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     tokenized_raw_text: True
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     train_split: None
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     use_vcs: True
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:   engine:
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     name: logger
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     nb_epochs: 120
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     print_freq: 10
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     saving_criteria: ['eval_epoch.metric.med_im2recipe_mean:min', 'eval_epoch.metric.recall_at_1_im2recipe_mean:max']
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:   exp:
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     checkpoint: None
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     dir: /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     resume: best_eval_epoch.metric.recall_at_1_im2recipe_mean
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:   misc:
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     cuda: True
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     data_parrallel: True
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     logs_name: None
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     overrite: False
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     seed: 1338
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:   model:
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     criterion:
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:         tri_cross_encoder: False
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       itm_loss_weight: 1
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       keep_background: False
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       retrieval_strategy:
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:         aggregation: valid
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:         get_tokens: True
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:         margin: 0.05
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:         margin_params:
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:           increment: 0.005
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:           increment_margin: True
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:           max_margin: 0.3
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:         name: triplet
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:         nb_samples: 9999
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:         sampling: max_negative
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:         substrategy: ['IRR', 'RII', 'SIRR', 'SRII']
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:         substrategy_weights: [1.0, 1.0, 0.1, 0.1]
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     import: recipe1m.models.factory
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     metric:
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       k_test: 10
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       nb_bags: 10
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       nb_matchs_per_bag: 1000
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       retrieval_dir: /tmp/recipe1m
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       save_ids: False
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       trijoint: True
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     name: trijoint
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     network:
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       activations: ['tanh', 'normalize']
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       aux_kwords: True
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       aux_kwords_encoder: True
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       avg_concat: True
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       bert_config: /home/mshukor/tfood/recipe1m/models/networks/recipe_networks/config_bert_albef.json
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       cat_pos: 0
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       checkpoint: None
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       class_attention: False
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       class_attention_fus: False
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       cls_token: False
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       cls_token_fus: False
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       cross_decoder_image: True
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       cross_decoder_recipe: True
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:         class_attention: False
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:         cls_token: False
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:         cross_attention: False
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:         cross_decoder: True
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:         double_decoder: False
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:         get_tokens: False
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:         load_backbones: False
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:         n_heads: 4
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:         n_layers: 4
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:         query_img: False
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       cross_transformer: False
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       dim_emb: 1024
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       dim_image_out: 2048
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       dim_ingr_out: 300
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       dim_instr_in: 1024
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       dim_instr_out: 1024
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       freeze_im: False
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       freeze_rec: False
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       get_tokens: True
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       get_tokens_cross_decoder_recipe: True
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       hidden_size: 512
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       image_backbone_name: clip
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       image_res: 224
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       kwords_same_level: False
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       last_cat_aux: True
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       n_heads: 4
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       n_heads_cross: 4
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       n_heads_cross_image: 4
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       n_heads_single: 4
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       n_layers: 2
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       n_layers_cross: 2
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       n_layers_cross_image: 1
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       n_layers_single: 2
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       num_hidden_layers_kw: 2
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       old: True
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       path_ingrs: /home/mshukor/data/recipe1m/text/vocab.pkl
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       path_vocab: /home/mshukor/data/recipe1m/text/vocab_all.txt
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       recipe_encoder: tfoodvicha-transformer
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       remove_additional_embedding: ['proj_image']
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       text_encoder: bert-base-uncased
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       vision_width: 768
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       vit_all_tokens: True
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       with_ingrs: True
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       with_instrs: True
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:       with_titles: True
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     with_classif: False
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:   optimizer:
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     batch_size_factor: None
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     clip_grad: 8.0
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     import: recipe1m.optimizers.factory
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     lr: 1e-05
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     lr_img: 1e-06
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     name: trijoint_fixed_fine_tune
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:     switch_epoch: 20
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:   path_opts: /home/mshukor/tfood/recipe1m/options/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_noimgproj.yaml
[S 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.75:   view: ['logs:train_epoch.loss', 'logs:train_epoch.bad_pairs', 'logs:eval_epoch.metric.med_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_1_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_5_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_10_im2recipe_mean', 'logs:eval_epoch.metric.med_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_1_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_5_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_10_recipe2im_mean', 'logs:optimizer.is_optimizer_recipe&image', 'logs:optimizer.total_norm']
[I 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.76: posix.uname_result(sysname='Linux', nodename='DGX', release='5.4.0-124-generic', version='#140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022', machine='x86_64')
[I 2022-10-28 13:43:37] ...ytorch/bootstrap/run.py.80: Available GPUs: [0]
[I 2022-10-28 13:43:37] ...strap/engines/factory.py.9: Creating engine...
[I 2022-10-28 13:43:37] ...trap/datasets/factory.py.8: Creating dataset...
[I 2022-10-28 13:43:39] ...m/datasets/recipe1m.py.152: Load VCs...
[I 2022-10-28 13:43:40] ...m/datasets/recipe1m.py.165: randkw_p... None
[I 2022-10-28 13:44:42] ...m/datasets/recipe1m.py.310: recipe elements to remove: [title,instructions]
[I 2022-10-28 13:44:42] ...rap/datasets/factory.py.21: Evaluation will take place on testset (51303 items)
[I 2022-10-28 13:44:42] ...strap/models/factory.py.21: Creating model...
[I 2022-10-28 13:44:43] ...e_networks/networks.py.203: Loading CLIP Visual model...
[I 2022-10-28 13:44:46] ...e_networks/networks.py.145: cuda
[I 2022-10-28 13:44:47] ...networks/tfood_vicha.py.49: Load VCs encoder...
[I 2022-10-28 13:44:50] ...els/metrics/trijoint.py.46: self.save_ids False
[I 2022-10-28 13:44:50] ...strap/models/factory.py.44: Enabling CUDA mode...
[I 2022-10-28 13:44:52] ...etworks/tfood_vicha.py.184: training aux kw modules
[I 2022-10-28 13:44:52] ...etworks/tfood_vicha.py.184: training aux kw modules
[I 2022-10-28 13:44:52] ...otstrap/views/factory.py.8: Creating views...
[I 2022-10-28 13:44:52] ...strap/engines/engine.py.98: Loading best_eval_epoch.metric.recall_at_1_im2recipe_mean checkpoint
[I 2022-10-28 13:44:52] ...trap/engines/engine.py.440: Loading model...
[I 2022-10-28 13:44:53] ...trap/engines/engine.py.453: Loading engine...
[I 2022-10-28 13:44:53] ...trap/engines/engine.py.108: Launching evaluation procedures
[I 2022-10-28 13:44:53] ...trap/engines/engine.py.268: Evaluating model on testset for epoch 116
[I 2022-10-28 13:44:53] ...m/datasets/recipe1m.py.400: Dataset will be sampled with "random" batch_sampler.
[I 2022-10-28 13:44:56] ...trap/engines/engine.py.321: eval: epoch 116 | batch 0/512
[I 2022-10-28 13:44:56] ...trap/engines/engine.py.325:       elapsed: 0:00:02 | left: 0:25:10
[I 2022-10-28 13:44:56] ...trap/engines/engine.py.326:       process: 2.95098 | load: 1.51532
[I 2022-10-28 13:44:59] ...trap/engines/engine.py.321: eval: epoch 116 | batch 10/512
[I 2022-10-28 13:44:59] ...trap/engines/engine.py.325:       elapsed: 0:00:05 | left: 0:04:49
[I 2022-10-28 13:44:59] ...trap/engines/engine.py.326:       process: 0.28741 | load: 0.00015
[I 2022-10-28 13:45:02] ...trap/engines/engine.py.321: eval: epoch 116 | batch 20/512
[I 2022-10-28 13:45:02] ...trap/engines/engine.py.325:       elapsed: 0:00:08 | left: 0:02:38
[I 2022-10-28 13:45:02] ...trap/engines/engine.py.326:       process: 0.28715 | load: 0.00016
[I 2022-10-28 13:45:05] ...trap/engines/engine.py.321: eval: epoch 116 | batch 30/512
[I 2022-10-28 13:45:05] ...trap/engines/engine.py.325:       elapsed: 0:00:11 | left: 0:02:23
[I 2022-10-28 13:45:05] ...trap/engines/engine.py.326:       process: 0.28567 | load: 0.00015
[I 2022-10-28 13:45:08] ...trap/engines/engine.py.321: eval: epoch 116 | batch 40/512
[I 2022-10-28 13:45:08] ...trap/engines/engine.py.325:       elapsed: 0:00:14 | left: 0:02:17
[I 2022-10-28 13:45:08] ...trap/engines/engine.py.326:       process: 0.29408 | load: 0.00016
[I 2022-10-28 13:45:11] ...trap/engines/engine.py.321: eval: epoch 116 | batch 50/512
[I 2022-10-28 13:45:11] ...trap/engines/engine.py.325:       elapsed: 0:00:17 | left: 0:02:15
[I 2022-10-28 13:45:11] ...trap/engines/engine.py.326:       process: 0.29136 | load: 0.00018
[I 2022-10-28 13:45:14] ...trap/engines/engine.py.321: eval: epoch 116 | batch 60/512
[I 2022-10-28 13:45:14] ...trap/engines/engine.py.325:       elapsed: 0:00:20 | left: 0:02:14
[I 2022-10-28 13:45:14] ...trap/engines/engine.py.326:       process: 0.30885 | load: 0.00016
[I 2022-10-28 13:45:17] ...trap/engines/engine.py.321: eval: epoch 116 | batch 70/512
[I 2022-10-28 13:45:17] ...trap/engines/engine.py.325:       elapsed: 0:00:23 | left: 0:02:10
[I 2022-10-28 13:45:17] ...trap/engines/engine.py.326:       process: 0.29298 | load: 0.00019
[I 2022-10-28 13:45:20] ...trap/engines/engine.py.321: eval: epoch 116 | batch 80/512
[I 2022-10-28 13:45:20] ...trap/engines/engine.py.325:       elapsed: 0:00:26 | left: 0:02:06
[I 2022-10-28 13:45:20] ...trap/engines/engine.py.326:       process: 0.29047 | load: 0.00018
[I 2022-10-28 13:45:23] ...trap/engines/engine.py.321: eval: epoch 116 | batch 90/512
[I 2022-10-28 13:45:23] ...trap/engines/engine.py.325:       elapsed: 0:00:29 | left: 0:02:02
[I 2022-10-28 13:45:23] ...trap/engines/engine.py.326:       process: 0.28559 | load: 0.00015
[I 2022-10-28 13:45:26] ...trap/engines/engine.py.321: eval: epoch 116 | batch 100/512
[I 2022-10-28 13:45:26] ...trap/engines/engine.py.325:       elapsed: 0:00:32 | left: 0:01:56
[I 2022-10-28 13:45:26] ...trap/engines/engine.py.326:       process: 0.25145 | load: 0.00016
[I 2022-10-28 13:45:28] ...trap/engines/engine.py.321: eval: epoch 116 | batch 110/512
[I 2022-10-28 13:45:28] ...trap/engines/engine.py.325:       elapsed: 0:00:34 | left: 0:01:42
[I 2022-10-28 13:45:28] ...trap/engines/engine.py.326:       process: 0.25441 | load: 0.00013
[I 2022-10-28 13:45:31] ...trap/engines/engine.py.321: eval: epoch 116 | batch 120/512
[I 2022-10-28 13:45:31] ...trap/engines/engine.py.325:       elapsed: 0:00:37 | left: 0:01:38
[I 2022-10-28 13:45:31] ...trap/engines/engine.py.326:       process: 0.25220 | load: 0.00014
[I 2022-10-28 13:45:33] ...trap/engines/engine.py.321: eval: epoch 116 | batch 130/512
[I 2022-10-28 13:45:33] ...trap/engines/engine.py.325:       elapsed: 0:00:39 | left: 0:01:36
[I 2022-10-28 13:45:33] ...trap/engines/engine.py.326:       process: 0.26638 | load: 0.00015
[I 2022-10-28 13:45:36] ...trap/engines/engine.py.321: eval: epoch 116 | batch 140/512
[I 2022-10-28 13:45:36] ...trap/engines/engine.py.325:       elapsed: 0:00:42 | left: 0:01:33
[I 2022-10-28 13:45:36] ...trap/engines/engine.py.326:       process: 0.25232 | load: 0.00016
[I 2022-10-28 13:45:38] ...trap/engines/engine.py.321: eval: epoch 116 | batch 150/512
[I 2022-10-28 13:45:38] ...trap/engines/engine.py.325:       elapsed: 0:00:44 | left: 0:01:31
[I 2022-10-28 13:45:38] ...trap/engines/engine.py.326:       process: 0.26324 | load: 0.00014
[I 2022-10-28 13:45:41] ...trap/engines/engine.py.321: eval: epoch 116 | batch 160/512
[I 2022-10-28 13:45:41] ...trap/engines/engine.py.325:       elapsed: 0:00:47 | left: 0:01:28
[I 2022-10-28 13:45:41] ...trap/engines/engine.py.326:       process: 0.25079 | load: 0.00023
[I 2022-10-28 13:45:43] ...trap/engines/engine.py.321: eval: epoch 116 | batch 170/512
[I 2022-10-28 13:45:43] ...trap/engines/engine.py.325:       elapsed: 0:00:49 | left: 0:01:25
[I 2022-10-28 13:45:43] ...trap/engines/engine.py.326:       process: 0.24877 | load: 0.00014
[I 2022-10-28 13:45:46] ...trap/engines/engine.py.321: eval: epoch 116 | batch 180/512
[I 2022-10-28 13:45:46] ...trap/engines/engine.py.325:       elapsed: 0:00:52 | left: 0:01:22
[I 2022-10-28 13:45:46] ...trap/engines/engine.py.326:       process: 0.24558 | load: 0.00014
[I 2022-10-28 13:45:48] ...trap/engines/engine.py.321: eval: epoch 116 | batch 190/512
[I 2022-10-28 13:45:48] ...trap/engines/engine.py.325:       elapsed: 0:00:54 | left: 0:01:20
[I 2022-10-28 13:45:48] ...trap/engines/engine.py.326:       process: 0.25115 | load: 0.00014
[I 2022-10-28 13:45:51] ...trap/engines/engine.py.321: eval: epoch 116 | batch 200/512
[I 2022-10-28 13:45:51] ...trap/engines/engine.py.325:       elapsed: 0:00:57 | left: 0:01:17
[I 2022-10-28 13:45:51] ...trap/engines/engine.py.326:       process: 0.24976 | load: 0.00015
[I 2022-10-28 13:45:53] ...trap/engines/engine.py.321: eval: epoch 116 | batch 210/512
[I 2022-10-28 13:45:53] ...trap/engines/engine.py.325:       elapsed: 0:00:59 | left: 0:01:15
[I 2022-10-28 13:45:53] ...trap/engines/engine.py.326:       process: 0.24801 | load: 0.00017
[I 2022-10-28 13:45:56] ...trap/engines/engine.py.321: eval: epoch 116 | batch 220/512
[I 2022-10-28 13:45:56] ...trap/engines/engine.py.325:       elapsed: 0:01:02 | left: 0:01:13
[I 2022-10-28 13:45:56] ...trap/engines/engine.py.326:       process: 0.25296 | load: 0.00015
[I 2022-10-28 13:45:58] ...trap/engines/engine.py.321: eval: epoch 116 | batch 230/512
[I 2022-10-28 13:45:58] ...trap/engines/engine.py.325:       elapsed: 0:01:04 | left: 0:01:10
[I 2022-10-28 13:45:58] ...trap/engines/engine.py.326:       process: 0.25309 | load: 0.00017
[I 2022-10-28 13:46:01] ...trap/engines/engine.py.321: eval: epoch 116 | batch 240/512
[I 2022-10-28 13:46:01] ...trap/engines/engine.py.325:       elapsed: 0:01:07 | left: 0:01:08
[I 2022-10-28 13:46:01] ...trap/engines/engine.py.326:       process: 0.25273 | load: 0.00015
[I 2022-10-28 13:46:03] ...trap/engines/engine.py.321: eval: epoch 116 | batch 250/512
[I 2022-10-28 13:46:03] ...trap/engines/engine.py.325:       elapsed: 0:01:09 | left: 0:01:05
[I 2022-10-28 13:46:03] ...trap/engines/engine.py.326:       process: 0.24906 | load: 0.00015
[I 2022-10-28 13:46:06] ...trap/engines/engine.py.321: eval: epoch 116 | batch 260/512
[I 2022-10-28 13:46:06] ...trap/engines/engine.py.325:       elapsed: 0:01:12 | left: 0:01:03
[I 2022-10-28 13:46:06] ...trap/engines/engine.py.326:       process: 0.25243 | load: 0.00014
[I 2022-10-28 13:46:08] ...trap/engines/engine.py.321: eval: epoch 116 | batch 270/512
[I 2022-10-28 13:46:08] ...trap/engines/engine.py.325:       elapsed: 0:01:14 | left: 0:01:00
[I 2022-10-28 13:46:08] ...trap/engines/engine.py.326:       process: 0.25351 | load: 0.00014
[I 2022-10-28 13:46:11] ...trap/engines/engine.py.321: eval: epoch 116 | batch 280/512
[I 2022-10-28 13:46:11] ...trap/engines/engine.py.325:       elapsed: 0:01:17 | left: 0:00:57
[I 2022-10-28 13:46:11] ...trap/engines/engine.py.326:       process: 0.25050 | load: 0.00014
[I 2022-10-28 13:46:13] ...trap/engines/engine.py.321: eval: epoch 116 | batch 290/512
[I 2022-10-28 13:46:13] ...trap/engines/engine.py.325:       elapsed: 0:01:19 | left: 0:00:55
[I 2022-10-28 13:46:13] ...trap/engines/engine.py.326:       process: 0.24239 | load: 0.00015
[I 2022-10-28 13:46:16] ...trap/engines/engine.py.321: eval: epoch 116 | batch 300/512
[I 2022-10-28 13:46:16] ...trap/engines/engine.py.325:       elapsed: 0:01:22 | left: 0:00:52
[I 2022-10-28 13:46:16] ...trap/engines/engine.py.326:       process: 0.24659 | load: 0.00015
[I 2022-10-28 13:46:18] ...trap/engines/engine.py.321: eval: epoch 116 | batch 310/512
[I 2022-10-28 13:46:18] ...trap/engines/engine.py.325:       elapsed: 0:01:24 | left: 0:00:50
[I 2022-10-28 13:46:18] ...trap/engines/engine.py.326:       process: 0.25423 | load: 0.00017
[I 2022-10-28 13:46:21] ...trap/engines/engine.py.321: eval: epoch 116 | batch 320/512
[I 2022-10-28 13:46:21] ...trap/engines/engine.py.325:       elapsed: 0:01:27 | left: 0:00:47
[I 2022-10-28 13:46:21] ...trap/engines/engine.py.326:       process: 0.24881 | load: 0.00014
[I 2022-10-28 13:46:23] ...trap/engines/engine.py.321: eval: epoch 116 | batch 330/512
[I 2022-10-28 13:46:23] ...trap/engines/engine.py.325:       elapsed: 0:01:29 | left: 0:00:45
[I 2022-10-28 13:46:23] ...trap/engines/engine.py.326:       process: 0.25273 | load: 0.00014
[I 2022-10-28 13:46:26] ...trap/engines/engine.py.321: eval: epoch 116 | batch 340/512
[I 2022-10-28 13:46:26] ...trap/engines/engine.py.325:       elapsed: 0:01:32 | left: 0:00:42
[I 2022-10-28 13:46:26] ...trap/engines/engine.py.326:       process: 0.25090 | load: 0.00014
[I 2022-10-28 13:46:28] ...trap/engines/engine.py.321: eval: epoch 116 | batch 350/512
[I 2022-10-28 13:46:28] ...trap/engines/engine.py.325:       elapsed: 0:01:34 | left: 0:00:40
[I 2022-10-28 13:46:28] ...trap/engines/engine.py.326:       process: 0.24984 | load: 0.00015
[I 2022-10-28 13:46:31] ...trap/engines/engine.py.321: eval: epoch 116 | batch 360/512
[I 2022-10-28 13:46:31] ...trap/engines/engine.py.325:       elapsed: 0:01:37 | left: 0:00:38
[I 2022-10-28 13:46:31] ...trap/engines/engine.py.326:       process: 0.25140 | load: 0.00014
[I 2022-10-28 13:46:33] ...trap/engines/engine.py.321: eval: epoch 116 | batch 370/512
[I 2022-10-28 13:46:33] ...trap/engines/engine.py.325:       elapsed: 0:01:39 | left: 0:00:35
[I 2022-10-28 13:46:33] ...trap/engines/engine.py.326:       process: 0.25254 | load: 0.00015
[I 2022-10-28 13:46:36] ...trap/engines/engine.py.321: eval: epoch 116 | batch 380/512
[I 2022-10-28 13:46:36] ...trap/engines/engine.py.325:       elapsed: 0:01:42 | left: 0:00:32
[I 2022-10-28 13:46:36] ...trap/engines/engine.py.326:       process: 0.25192 | load: 0.00016
[I 2022-10-28 13:46:38] ...trap/engines/engine.py.321: eval: epoch 116 | batch 390/512
[I 2022-10-28 13:46:38] ...trap/engines/engine.py.325:       elapsed: 0:01:44 | left: 0:00:30
[I 2022-10-28 13:46:38] ...trap/engines/engine.py.326:       process: 0.24997 | load: 0.00015
[I 2022-10-28 13:46:41] ...trap/engines/engine.py.321: eval: epoch 116 | batch 400/512
[I 2022-10-28 13:46:41] ...trap/engines/engine.py.325:       elapsed: 0:01:47 | left: 0:00:27
[I 2022-10-28 13:46:41] ...trap/engines/engine.py.326:       process: 0.24892 | load: 0.00015
[I 2022-10-28 13:46:43] ...trap/engines/engine.py.321: eval: epoch 116 | batch 410/512
[I 2022-10-28 13:46:43] ...trap/engines/engine.py.325:       elapsed: 0:01:49 | left: 0:00:25
[I 2022-10-28 13:46:43] ...trap/engines/engine.py.326:       process: 0.25359 | load: 0.00014
[I 2022-10-28 13:46:46] ...trap/engines/engine.py.321: eval: epoch 116 | batch 420/512
[I 2022-10-28 13:46:46] ...trap/engines/engine.py.325:       elapsed: 0:01:52 | left: 0:00:23
[I 2022-10-28 13:46:46] ...trap/engines/engine.py.326:       process: 0.25100 | load: 0.00018
[I 2022-10-28 13:46:48] ...trap/engines/engine.py.321: eval: epoch 116 | batch 430/512
[I 2022-10-28 13:46:48] ...trap/engines/engine.py.325:       elapsed: 0:01:54 | left: 0:00:20
[I 2022-10-28 13:46:48] ...trap/engines/engine.py.326:       process: 0.25242 | load: 0.00015
[I 2022-10-28 13:46:51] ...trap/engines/engine.py.321: eval: epoch 116 | batch 440/512
[I 2022-10-28 13:46:51] ...trap/engines/engine.py.325:       elapsed: 0:01:57 | left: 0:00:18
[I 2022-10-28 13:46:51] ...trap/engines/engine.py.326:       process: 0.25231 | load: 0.00015
[I 2022-10-28 13:46:53] ...trap/engines/engine.py.321: eval: epoch 116 | batch 450/512
[I 2022-10-28 13:46:53] ...trap/engines/engine.py.325:       elapsed: 0:01:59 | left: 0:00:15
[I 2022-10-28 13:46:53] ...trap/engines/engine.py.326:       process: 0.24976 | load: 0.00015
[I 2022-10-28 13:46:56] ...trap/engines/engine.py.321: eval: epoch 116 | batch 460/512
[I 2022-10-28 13:46:56] ...trap/engines/engine.py.325:       elapsed: 0:02:02 | left: 0:00:12
[I 2022-10-28 13:46:56] ...trap/engines/engine.py.326:       process: 0.25419 | load: 0.00014
[I 2022-10-28 13:46:58] ...trap/engines/engine.py.321: eval: epoch 116 | batch 470/512
[I 2022-10-28 13:46:58] ...trap/engines/engine.py.325:       elapsed: 0:02:04 | left: 0:00:10
[I 2022-10-28 13:46:58] ...trap/engines/engine.py.326:       process: 0.24783 | load: 0.00015
[I 2022-10-28 13:47:01] ...trap/engines/engine.py.321: eval: epoch 116 | batch 480/512
[I 2022-10-28 13:47:01] ...trap/engines/engine.py.325:       elapsed: 0:02:07 | left: 0:00:07
[I 2022-10-28 13:47:01] ...trap/engines/engine.py.326:       process: 0.24613 | load: 0.00014
[I 2022-10-28 13:47:03] ...trap/engines/engine.py.321: eval: epoch 116 | batch 490/512
[I 2022-10-28 13:47:03] ...trap/engines/engine.py.325:       elapsed: 0:02:09 | left: 0:00:05
[I 2022-10-28 13:47:03] ...trap/engines/engine.py.326:       process: 0.24277 | load: 0.00015
[I 2022-10-28 13:47:06] ...trap/engines/engine.py.321: eval: epoch 116 | batch 500/512
[I 2022-10-28 13:47:06] ...trap/engines/engine.py.325:       elapsed: 0:02:12 | left: 0:00:02
[I 2022-10-28 13:47:06] ...trap/engines/engine.py.326:       process: 0.25042 | load: 0.00014
[I 2022-10-28 13:47:08] ...trap/engines/engine.py.321: eval: epoch 116 | batch 510/512
[I 2022-10-28 13:47:08] ...trap/engines/engine.py.325:       elapsed: 0:02:14 | left: 0:00:00
[I 2022-10-28 13:47:08] ...trap/engines/engine.py.326:       process: 0.24125 | load: 0.00010
[S 2022-10-28 13:47:09] ...trap/engines/engine.py.113: eval_epoch.epoch: 116
[I 2022-10-28 13:47:09] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs
[S 2022-10-28 13:47:13] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 2.00
[S 2022-10-28 13:47:13] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.20
[S 2022-10-28 13:47:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.429200
[S 2022-10-28 13:47:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.696400
[S 2022-10-28 13:47:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.780900
[S 2022-10-28 13:47:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.509600
[S 2022-10-28 13:47:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.768600
[S 2022-10-28 13:47:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.836300
[S 2022-10-28 13:47:13] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000
[S 2022-10-28 13:47:13] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.400000
[S 2022-10-28 13:47:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.009796
[S 2022-10-28 13:47:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.006621
[S 2022-10-28 13:47:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.009904
[S 2022-10-28 13:47:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.009666
[S 2022-10-28 13:47:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.010509
[S 2022-10-28 13:47:13] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.011782
[I 2022-10-28 13:47:14] ...trap/engines/engine.py.115: Ending evaluation procedures
[W 2022-10-28 13:47:14] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json'
[W 2022-10-28 13:47:14] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[W 2022-10-28 13:47:14] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[I 2022-10-28 13:47:18] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html
[I 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.73: Saving environment info
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75: options: 
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:   dataset:
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     aux_kw_path:
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       test: /home/mshukor/data/recipe1m/clip_da/layer1_test_titles_kw.json
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       train: /home/mshukor/data/recipe1m/clip_da/layer1_train_titles_kw.json
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       val: /home/mshukor/data/recipe1m/clip_da/layer1_val_titles_kw.json
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     batch_sampler: triplet_classif
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     batch_size: 100
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     database: lmdb
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     dir: /home/mshukor/data/recipe1m
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     eval_split: test
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     freq_mismatch: 0.0
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     image_from: database
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     import: recipe1m.datasets.factory
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     interchange_ingrd_instr: None
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     kw_path:
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       test: /home/mshukor/data/recipe1m/clip_da/layer1_test_ingr_kw.json
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       train: /home/mshukor/data/recipe1m/clip_da/layer1_train_ingr_kw.json
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       val: /home/mshukor/data/recipe1m/clip_da/layer1_val_ingr_kw.json
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     name: recipe1m
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     nb_classes: 1048
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     nb_threads: 4
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     randkw_p: 0.3
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     randkw_p_aux: 0.5
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     remove_list: [ingredients,instructions]
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     tokenized_raw_text: True
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     train_split: None
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     use_vcs: True
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:   engine:
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     name: logger
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     nb_epochs: 120
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     print_freq: 10
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     saving_criteria: ['eval_epoch.metric.med_im2recipe_mean:min', 'eval_epoch.metric.recall_at_1_im2recipe_mean:max']
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:   exp:
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     checkpoint: None
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     dir: /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     resume: best_eval_epoch.metric.recall_at_1_im2recipe_mean
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:   misc:
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     cuda: True
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     data_parrallel: True
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     logs_name: None
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     overrite: False
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     seed: 1338
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:   model:
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     criterion:
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:         tri_cross_encoder: False
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       itm_loss_weight: 1
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       keep_background: False
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       retrieval_strategy:
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:         aggregation: valid
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:         get_tokens: True
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:         margin: 0.05
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:         margin_params:
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:           increment: 0.005
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:           increment_margin: True
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:           max_margin: 0.3
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:         name: triplet
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:         nb_samples: 9999
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:         sampling: max_negative
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:         substrategy: ['IRR', 'RII', 'SIRR', 'SRII']
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:         substrategy_weights: [1.0, 1.0, 0.1, 0.1]
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     import: recipe1m.models.factory
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     metric:
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       k_test: 10
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       nb_bags: 10
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       nb_matchs_per_bag: 1000
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       retrieval_dir: /tmp/recipe1m
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       save_ids: False
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       trijoint: True
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     name: trijoint
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     network:
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       activations: ['tanh', 'normalize']
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       aux_kwords: True
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       aux_kwords_encoder: True
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       avg_concat: True
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       bert_config: /home/mshukor/tfood/recipe1m/models/networks/recipe_networks/config_bert_albef.json
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       cat_pos: 0
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       checkpoint: None
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       class_attention: False
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       class_attention_fus: False
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       cls_token: False
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       cls_token_fus: False
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       cross_decoder_image: True
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       cross_decoder_recipe: True
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:         class_attention: False
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:         cls_token: False
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:         cross_attention: False
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:         cross_decoder: True
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:         double_decoder: False
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:         get_tokens: False
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:         load_backbones: False
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:         n_heads: 4
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:         n_layers: 4
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:         query_img: False
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       cross_transformer: False
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       dim_emb: 1024
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       dim_image_out: 2048
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       dim_ingr_out: 300
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       dim_instr_in: 1024
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       dim_instr_out: 1024
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       freeze_im: False
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       freeze_rec: False
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       get_tokens: True
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       get_tokens_cross_decoder_recipe: True
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       hidden_size: 512
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       image_backbone_name: clip
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       image_res: 224
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       kwords_same_level: False
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       last_cat_aux: True
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       n_heads: 4
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       n_heads_cross: 4
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       n_heads_cross_image: 4
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       n_heads_single: 4
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       n_layers: 2
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       n_layers_cross: 2
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       n_layers_cross_image: 1
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       n_layers_single: 2
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       num_hidden_layers_kw: 2
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       old: True
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       path_ingrs: /home/mshukor/data/recipe1m/text/vocab.pkl
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       path_vocab: /home/mshukor/data/recipe1m/text/vocab_all.txt
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       recipe_encoder: tfoodvicha-transformer
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       remove_additional_embedding: ['proj_image']
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       text_encoder: bert-base-uncased
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       vision_width: 768
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       vit_all_tokens: True
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       with_ingrs: True
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       with_instrs: True
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:       with_titles: True
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     with_classif: False
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:   optimizer:
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     batch_size_factor: None
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     clip_grad: 8.0
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     import: recipe1m.optimizers.factory
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     lr: 1e-05
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     lr_img: 1e-06
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     name: trijoint_fixed_fine_tune
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:     switch_epoch: 20
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:   path_opts: /home/mshukor/tfood/recipe1m/options/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_noimgproj.yaml
[S 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.75:   view: ['logs:train_epoch.loss', 'logs:train_epoch.bad_pairs', 'logs:eval_epoch.metric.med_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_1_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_5_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_10_im2recipe_mean', 'logs:eval_epoch.metric.med_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_1_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_5_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_10_recipe2im_mean', 'logs:optimizer.is_optimizer_recipe&image', 'logs:optimizer.total_norm']
[I 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.76: posix.uname_result(sysname='Linux', nodename='DGX', release='5.4.0-124-generic', version='#140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022', machine='x86_64')
[I 2022-10-28 14:08:13] ...ytorch/bootstrap/run.py.80: Available GPUs: [0]
[I 2022-10-28 14:08:13] ...strap/engines/factory.py.9: Creating engine...
[I 2022-10-28 14:08:13] ...trap/datasets/factory.py.8: Creating dataset...
[I 2022-10-28 14:08:15] ...m/datasets/recipe1m.py.152: Load VCs...
[I 2022-10-28 14:08:16] ...m/datasets/recipe1m.py.165: randkw_p... None
[I 2022-10-28 14:09:19] ...m/datasets/recipe1m.py.310: recipe elements to remove: [ingredients,instructions]
[I 2022-10-28 14:09:19] ...rap/datasets/factory.py.21: Evaluation will take place on testset (51303 items)
[I 2022-10-28 14:09:19] ...strap/models/factory.py.21: Creating model...
[I 2022-10-28 14:09:20] ...e_networks/networks.py.203: Loading CLIP Visual model...
[I 2022-10-28 14:09:23] ...e_networks/networks.py.145: cuda
[I 2022-10-28 14:09:24] ...networks/tfood_vicha.py.49: Load VCs encoder...
[I 2022-10-28 14:09:27] ...els/metrics/trijoint.py.46: self.save_ids False
[I 2022-10-28 14:09:27] ...strap/models/factory.py.44: Enabling CUDA mode...
[I 2022-10-28 14:09:29] ...etworks/tfood_vicha.py.184: training aux kw modules
[I 2022-10-28 14:09:29] ...etworks/tfood_vicha.py.184: training aux kw modules
[I 2022-10-28 14:09:29] ...otstrap/views/factory.py.8: Creating views...
[I 2022-10-28 14:09:29] ...strap/engines/engine.py.98: Loading best_eval_epoch.metric.recall_at_1_im2recipe_mean checkpoint
[I 2022-10-28 14:09:29] ...trap/engines/engine.py.440: Loading model...
[I 2022-10-28 14:09:30] ...trap/engines/engine.py.453: Loading engine...
[I 2022-10-28 14:09:30] ...trap/engines/engine.py.108: Launching evaluation procedures
[I 2022-10-28 14:09:30] ...trap/engines/engine.py.268: Evaluating model on testset for epoch 116
[I 2022-10-28 14:09:30] ...m/datasets/recipe1m.py.400: Dataset will be sampled with "random" batch_sampler.
[I 2022-10-28 14:09:33] ...trap/engines/engine.py.321: eval: epoch 116 | batch 0/512
[I 2022-10-28 14:09:33] ...trap/engines/engine.py.325:       elapsed: 0:00:02 | left: 0:23:08
[I 2022-10-28 14:09:33] ...trap/engines/engine.py.326:       process: 2.71237 | load: 1.47971
[I 2022-10-28 14:09:36] ...trap/engines/engine.py.321: eval: epoch 116 | batch 10/512
[I 2022-10-28 14:09:36] ...trap/engines/engine.py.325:       elapsed: 0:00:05 | left: 0:04:30
[I 2022-10-28 14:09:36] ...trap/engines/engine.py.326:       process: 0.27468 | load: 0.00014
[I 2022-10-28 14:09:39] ...trap/engines/engine.py.321: eval: epoch 116 | batch 20/512
[I 2022-10-28 14:09:39] ...trap/engines/engine.py.325:       elapsed: 0:00:08 | left: 0:02:30
[I 2022-10-28 14:09:39] ...trap/engines/engine.py.326:       process: 0.27267 | load: 0.00014
[I 2022-10-28 14:09:41] ...trap/engines/engine.py.321: eval: epoch 116 | batch 30/512
[I 2022-10-28 14:09:41] ...trap/engines/engine.py.325:       elapsed: 0:00:10 | left: 0:02:12
[I 2022-10-28 14:09:41] ...trap/engines/engine.py.326:       process: 0.27343 | load: 0.00014
[I 2022-10-28 14:09:44] ...trap/engines/engine.py.321: eval: epoch 116 | batch 40/512
[I 2022-10-28 14:09:44] ...trap/engines/engine.py.325:       elapsed: 0:00:13 | left: 0:02:07
[I 2022-10-28 14:09:44] ...trap/engines/engine.py.326:       process: 0.27272 | load: 0.00014
[I 2022-10-28 14:09:47] ...trap/engines/engine.py.321: eval: epoch 116 | batch 50/512
[I 2022-10-28 14:09:47] ...trap/engines/engine.py.325:       elapsed: 0:00:16 | left: 0:02:09
[I 2022-10-28 14:09:47] ...trap/engines/engine.py.326:       process: 0.27318 | load: 0.00015
[I 2022-10-28 14:09:50] ...trap/engines/engine.py.321: eval: epoch 116 | batch 60/512
[I 2022-10-28 14:09:50] ...trap/engines/engine.py.325:       elapsed: 0:00:19 | left: 0:02:04
[I 2022-10-28 14:09:50] ...trap/engines/engine.py.326:       process: 0.27321 | load: 0.00016
[I 2022-10-28 14:09:52] ...trap/engines/engine.py.321: eval: epoch 116 | batch 70/512
[I 2022-10-28 14:09:52] ...trap/engines/engine.py.325:       elapsed: 0:00:22 | left: 0:02:02
[I 2022-10-28 14:09:52] ...trap/engines/engine.py.326:       process: 0.27306 | load: 0.00016
[I 2022-10-28 14:09:55] ...trap/engines/engine.py.321: eval: epoch 116 | batch 80/512
[I 2022-10-28 14:09:55] ...trap/engines/engine.py.325:       elapsed: 0:00:24 | left: 0:01:58
[I 2022-10-28 14:09:55] ...trap/engines/engine.py.326:       process: 0.27262 | load: 0.00019
[I 2022-10-28 14:09:58] ...trap/engines/engine.py.321: eval: epoch 116 | batch 90/512
[I 2022-10-28 14:09:58] ...trap/engines/engine.py.325:       elapsed: 0:00:27 | left: 0:01:55
[I 2022-10-28 14:09:58] ...trap/engines/engine.py.326:       process: 0.27159 | load: 0.00016
[I 2022-10-28 14:10:01] ...trap/engines/engine.py.321: eval: epoch 116 | batch 100/512
[I 2022-10-28 14:10:01] ...trap/engines/engine.py.325:       elapsed: 0:00:30 | left: 0:01:48
[I 2022-10-28 14:10:01] ...trap/engines/engine.py.326:       process: 0.23181 | load: 0.00016
[I 2022-10-28 14:10:03] ...trap/engines/engine.py.321: eval: epoch 116 | batch 110/512
[I 2022-10-28 14:10:03] ...trap/engines/engine.py.325:       elapsed: 0:00:32 | left: 0:01:34
[I 2022-10-28 14:10:03] ...trap/engines/engine.py.326:       process: 0.23283 | load: 0.00015
[I 2022-10-28 14:10:05] ...trap/engines/engine.py.321: eval: epoch 116 | batch 120/512
[I 2022-10-28 14:10:05] ...trap/engines/engine.py.325:       elapsed: 0:00:34 | left: 0:01:31
[I 2022-10-28 14:10:05] ...trap/engines/engine.py.326:       process: 0.23583 | load: 0.00013
[I 2022-10-28 14:10:08] ...trap/engines/engine.py.321: eval: epoch 116 | batch 130/512
[I 2022-10-28 14:10:08] ...trap/engines/engine.py.325:       elapsed: 0:00:37 | left: 0:01:28
[I 2022-10-28 14:10:08] ...trap/engines/engine.py.326:       process: 0.23369 | load: 0.00014
[I 2022-10-28 14:10:10] ...trap/engines/engine.py.321: eval: epoch 116 | batch 140/512
[I 2022-10-28 14:10:10] ...trap/engines/engine.py.325:       elapsed: 0:00:39 | left: 0:01:26
[I 2022-10-28 14:10:10] ...trap/engines/engine.py.326:       process: 0.23139 | load: 0.00014
[I 2022-10-28 14:10:12] ...trap/engines/engine.py.321: eval: epoch 116 | batch 150/512
[I 2022-10-28 14:10:12] ...trap/engines/engine.py.325:       elapsed: 0:00:41 | left: 0:01:24
[I 2022-10-28 14:10:12] ...trap/engines/engine.py.326:       process: 0.23221 | load: 0.00014
[I 2022-10-28 14:10:15] ...trap/engines/engine.py.321: eval: epoch 116 | batch 160/512
[I 2022-10-28 14:10:15] ...trap/engines/engine.py.325:       elapsed: 0:00:44 | left: 0:01:21
[I 2022-10-28 14:10:15] ...trap/engines/engine.py.326:       process: 0.23200 | load: 0.00015
[I 2022-10-28 14:10:17] ...trap/engines/engine.py.321: eval: epoch 116 | batch 170/512
[I 2022-10-28 14:10:17] ...trap/engines/engine.py.325:       elapsed: 0:00:46 | left: 0:01:19
[I 2022-10-28 14:10:17] ...trap/engines/engine.py.326:       process: 0.23142 | load: 0.00014
[I 2022-10-28 14:10:19] ...trap/engines/engine.py.321: eval: epoch 116 | batch 180/512
[I 2022-10-28 14:10:19] ...trap/engines/engine.py.325:       elapsed: 0:00:48 | left: 0:01:17
[I 2022-10-28 14:10:19] ...trap/engines/engine.py.326:       process: 0.23357 | load: 0.00016
[I 2022-10-28 14:10:22] ...trap/engines/engine.py.321: eval: epoch 116 | batch 190/512
[I 2022-10-28 14:10:22] ...trap/engines/engine.py.325:       elapsed: 0:00:51 | left: 0:01:14
[I 2022-10-28 14:10:22] ...trap/engines/engine.py.326:       process: 0.23441 | load: 0.00014
[I 2022-10-28 14:10:24] ...trap/engines/engine.py.321: eval: epoch 116 | batch 200/512
[I 2022-10-28 14:10:24] ...trap/engines/engine.py.325:       elapsed: 0:00:53 | left: 0:01:12
[I 2022-10-28 14:10:24] ...trap/engines/engine.py.326:       process: 0.23353 | load: 0.00014
[I 2022-10-28 14:10:26] ...trap/engines/engine.py.321: eval: epoch 116 | batch 210/512
[I 2022-10-28 14:10:26] ...trap/engines/engine.py.325:       elapsed: 0:00:55 | left: 0:01:09
[I 2022-10-28 14:10:26] ...trap/engines/engine.py.326:       process: 0.22721 | load: 0.00015
[I 2022-10-28 14:10:29] ...trap/engines/engine.py.321: eval: epoch 116 | batch 220/512
[I 2022-10-28 14:10:29] ...trap/engines/engine.py.325:       elapsed: 0:00:58 | left: 0:01:07
[I 2022-10-28 14:10:29] ...trap/engines/engine.py.326:       process: 0.23205 | load: 0.00015
[I 2022-10-28 14:10:31] ...trap/engines/engine.py.321: eval: epoch 116 | batch 230/512
[I 2022-10-28 14:10:31] ...trap/engines/engine.py.325:       elapsed: 0:01:00 | left: 0:01:05
[I 2022-10-28 14:10:31] ...trap/engines/engine.py.326:       process: 0.23344 | load: 0.00014
[I 2022-10-28 14:10:33] ...trap/engines/engine.py.321: eval: epoch 116 | batch 240/512
[I 2022-10-28 14:10:33] ...trap/engines/engine.py.325:       elapsed: 0:01:02 | left: 0:01:03
[I 2022-10-28 14:10:33] ...trap/engines/engine.py.326:       process: 0.23337 | load: 0.00014
[I 2022-10-28 14:10:36] ...trap/engines/engine.py.321: eval: epoch 116 | batch 250/512
[I 2022-10-28 14:10:36] ...trap/engines/engine.py.325:       elapsed: 0:01:05 | left: 0:01:00
[I 2022-10-28 14:10:36] ...trap/engines/engine.py.326:       process: 0.23124 | load: 0.00016
[I 2022-10-28 14:10:38] ...trap/engines/engine.py.321: eval: epoch 116 | batch 260/512
[I 2022-10-28 14:10:38] ...trap/engines/engine.py.325:       elapsed: 0:01:07 | left: 0:00:58
[I 2022-10-28 14:10:38] ...trap/engines/engine.py.326:       process: 0.23331 | load: 0.00016
[I 2022-10-28 14:10:40] ...trap/engines/engine.py.321: eval: epoch 116 | batch 270/512
[I 2022-10-28 14:10:40] ...trap/engines/engine.py.325:       elapsed: 0:01:09 | left: 0:00:55
[I 2022-10-28 14:10:40] ...trap/engines/engine.py.326:       process: 0.22759 | load: 0.00015
[I 2022-10-28 14:10:42] ...trap/engines/engine.py.321: eval: epoch 116 | batch 280/512
[I 2022-10-28 14:10:42] ...trap/engines/engine.py.325:       elapsed: 0:01:12 | left: 0:00:53
[I 2022-10-28 14:10:42] ...trap/engines/engine.py.326:       process: 0.23083 | load: 0.00016
[I 2022-10-28 14:10:45] ...trap/engines/engine.py.321: eval: epoch 116 | batch 290/512
[I 2022-10-28 14:10:45] ...trap/engines/engine.py.325:       elapsed: 0:01:14 | left: 0:00:51
[I 2022-10-28 14:10:45] ...trap/engines/engine.py.326:       process: 0.23190 | load: 0.00014
[I 2022-10-28 14:10:47] ...trap/engines/engine.py.321: eval: epoch 116 | batch 300/512
[I 2022-10-28 14:10:47] ...trap/engines/engine.py.325:       elapsed: 0:01:16 | left: 0:00:49
[I 2022-10-28 14:10:47] ...trap/engines/engine.py.326:       process: 0.23247 | load: 0.00023
[I 2022-10-28 14:10:49] ...trap/engines/engine.py.321: eval: epoch 116 | batch 310/512
[I 2022-10-28 14:10:49] ...trap/engines/engine.py.325:       elapsed: 0:01:19 | left: 0:00:47
[I 2022-10-28 14:10:49] ...trap/engines/engine.py.326:       process: 0.23047 | load: 0.00014
[I 2022-10-28 14:10:52] ...trap/engines/engine.py.321: eval: epoch 116 | batch 320/512
[I 2022-10-28 14:10:52] ...trap/engines/engine.py.325:       elapsed: 0:01:21 | left: 0:00:44
[I 2022-10-28 14:10:52] ...trap/engines/engine.py.326:       process: 0.22800 | load: 0.00016
[I 2022-10-28 14:10:54] ...trap/engines/engine.py.321: eval: epoch 116 | batch 330/512
[I 2022-10-28 14:10:54] ...trap/engines/engine.py.325:       elapsed: 0:01:23 | left: 0:00:43
[I 2022-10-28 14:10:54] ...trap/engines/engine.py.326:       process: 0.23379 | load: 0.00014
[I 2022-10-28 14:10:57] ...trap/engines/engine.py.321: eval: epoch 116 | batch 340/512
[I 2022-10-28 14:10:57] ...trap/engines/engine.py.325:       elapsed: 0:01:26 | left: 0:00:40
[I 2022-10-28 14:10:57] ...trap/engines/engine.py.326:       process: 0.23175 | load: 0.00015
[I 2022-10-28 14:10:59] ...trap/engines/engine.py.321: eval: epoch 116 | batch 350/512
[I 2022-10-28 14:10:59] ...trap/engines/engine.py.325:       elapsed: 0:01:28 | left: 0:00:37
[I 2022-10-28 14:10:59] ...trap/engines/engine.py.326:       process: 0.23269 | load: 0.00015
[I 2022-10-28 14:11:01] ...trap/engines/engine.py.321: eval: epoch 116 | batch 360/512
[I 2022-10-28 14:11:01] ...trap/engines/engine.py.325:       elapsed: 0:01:30 | left: 0:00:35
[I 2022-10-28 14:11:01] ...trap/engines/engine.py.326:       process: 0.23316 | load: 0.00016
[I 2022-10-28 14:11:04] ...trap/engines/engine.py.321: eval: epoch 116 | batch 370/512
[I 2022-10-28 14:11:04] ...trap/engines/engine.py.325:       elapsed: 0:01:33 | left: 0:00:32
[I 2022-10-28 14:11:04] ...trap/engines/engine.py.326:       process: 0.23329 | load: 0.00016
[I 2022-10-28 14:11:06] ...trap/engines/engine.py.321: eval: epoch 116 | batch 380/512
[I 2022-10-28 14:11:06] ...trap/engines/engine.py.325:       elapsed: 0:01:35 | left: 0:00:30
[I 2022-10-28 14:11:06] ...trap/engines/engine.py.326:       process: 0.23096 | load: 0.00016
[I 2022-10-28 14:11:08] ...trap/engines/engine.py.321: eval: epoch 116 | batch 390/512
[I 2022-10-28 14:11:08] ...trap/engines/engine.py.325:       elapsed: 0:01:37 | left: 0:00:28
[I 2022-10-28 14:11:08] ...trap/engines/engine.py.326:       process: 0.23214 | load: 0.00016
[I 2022-10-28 14:11:11] ...trap/engines/engine.py.321: eval: epoch 116 | batch 400/512
[I 2022-10-28 14:11:11] ...trap/engines/engine.py.325:       elapsed: 0:01:40 | left: 0:00:26
[I 2022-10-28 14:11:11] ...trap/engines/engine.py.326:       process: 0.23364 | load: 0.00016
[I 2022-10-28 14:11:13] ...trap/engines/engine.py.321: eval: epoch 116 | batch 410/512
[I 2022-10-28 14:11:13] ...trap/engines/engine.py.325:       elapsed: 0:01:42 | left: 0:00:23
[I 2022-10-28 14:11:13] ...trap/engines/engine.py.326:       process: 0.23241 | load: 0.00017
[I 2022-10-28 14:11:15] ...trap/engines/engine.py.321: eval: epoch 116 | batch 420/512
[I 2022-10-28 14:11:15] ...trap/engines/engine.py.325:       elapsed: 0:01:44 | left: 0:00:21
[I 2022-10-28 14:11:15] ...trap/engines/engine.py.326:       process: 0.23219 | load: 0.00015
[I 2022-10-28 14:11:17] ...trap/engines/engine.py.321: eval: epoch 116 | batch 430/512
[I 2022-10-28 14:11:17] ...trap/engines/engine.py.325:       elapsed: 0:01:47 | left: 0:00:19
[I 2022-10-28 14:11:18] ...trap/engines/engine.py.326:       process: 0.23202 | load: 0.00016
[I 2022-10-28 14:11:20] ...trap/engines/engine.py.321: eval: epoch 116 | batch 440/512
[I 2022-10-28 14:11:20] ...trap/engines/engine.py.325:       elapsed: 0:01:49 | left: 0:00:16
[I 2022-10-28 14:11:20] ...trap/engines/engine.py.326:       process: 0.23007 | load: 0.00016
[I 2022-10-28 14:11:22] ...trap/engines/engine.py.321: eval: epoch 116 | batch 450/512
[I 2022-10-28 14:11:22] ...trap/engines/engine.py.325:       elapsed: 0:01:51 | left: 0:00:14
[I 2022-10-28 14:11:22] ...trap/engines/engine.py.326:       process: 0.23088 | load: 0.00017
[I 2022-10-28 14:11:24] ...trap/engines/engine.py.321: eval: epoch 116 | batch 460/512
[I 2022-10-28 14:11:24] ...trap/engines/engine.py.325:       elapsed: 0:01:54 | left: 0:00:12
[I 2022-10-28 14:11:24] ...trap/engines/engine.py.326:       process: 0.23252 | load: 0.00016
[I 2022-10-28 14:11:27] ...trap/engines/engine.py.321: eval: epoch 116 | batch 470/512
[I 2022-10-28 14:11:27] ...trap/engines/engine.py.325:       elapsed: 0:01:56 | left: 0:00:09
[I 2022-10-28 14:11:27] ...trap/engines/engine.py.326:       process: 0.23200 | load: 0.00016
[I 2022-10-28 14:11:29] ...trap/engines/engine.py.321: eval: epoch 116 | batch 480/512
[I 2022-10-28 14:11:29] ...trap/engines/engine.py.325:       elapsed: 0:01:58 | left: 0:00:07
[I 2022-10-28 14:11:29] ...trap/engines/engine.py.326:       process: 0.23243 | load: 0.00016
[I 2022-10-28 14:11:31] ...trap/engines/engine.py.321: eval: epoch 116 | batch 490/512
[I 2022-10-28 14:11:31] ...trap/engines/engine.py.325:       elapsed: 0:02:01 | left: 0:00:05
[I 2022-10-28 14:11:31] ...trap/engines/engine.py.326:       process: 0.23039 | load: 0.00018
[I 2022-10-28 14:11:34] ...trap/engines/engine.py.321: eval: epoch 116 | batch 500/512
[I 2022-10-28 14:11:34] ...trap/engines/engine.py.325:       elapsed: 0:02:03 | left: 0:00:02
[I 2022-10-28 14:11:34] ...trap/engines/engine.py.326:       process: 0.23181 | load: 0.00016
[I 2022-10-28 14:11:36] ...trap/engines/engine.py.321: eval: epoch 116 | batch 510/512
[I 2022-10-28 14:11:36] ...trap/engines/engine.py.325:       elapsed: 0:02:05 | left: 0:00:00
[I 2022-10-28 14:11:36] ...trap/engines/engine.py.326:       process: 0.22898 | load: 0.00010
[S 2022-10-28 14:11:37] ...trap/engines/engine.py.113: eval_epoch.epoch: 116
[I 2022-10-28 14:11:37] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs
[S 2022-10-28 14:11:41] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 68.10
[S 2022-10-28 14:11:41] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 11.05
[S 2022-10-28 14:11:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.057800
[S 2022-10-28 14:11:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.156200
[S 2022-10-28 14:11:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.228200
[S 2022-10-28 14:11:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.170200
[S 2022-10-28 14:11:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.388100
[S 2022-10-28 14:11:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.489400
[S 2022-10-28 14:11:41] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 5.48
[S 2022-10-28 14:11:41] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.788987
[S 2022-10-28 14:11:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.006274
[S 2022-10-28 14:11:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009908
[S 2022-10-28 14:11:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.014518
[S 2022-10-28 14:11:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.010332
[S 2022-10-28 14:11:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.013685
[S 2022-10-28 14:11:41] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.013559
[I 2022-10-28 14:11:42] ...trap/engines/engine.py.115: Ending evaluation procedures
[W 2022-10-28 14:11:42] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json'
[W 2022-10-28 14:11:42] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[W 2022-10-28 14:11:42] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[I 2022-10-28 14:11:46] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html
[I 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.73: Saving environment info
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75: options: 
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:   dataset:
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:     aux_kw_path:
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:       test: /home/mshukor/data/recipe1m/clip_da/layer1_test_titles_kw.json
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:       train: /home/mshukor/data/recipe1m/clip_da/layer1_train_titles_kw.json
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:       val: /home/mshukor/data/recipe1m/clip_da/layer1_val_titles_kw.json
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:     batch_sampler: triplet_classif
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:     batch_size: 100
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:     database: lmdb
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:     dir: /home/mshukor/data/recipe1m
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:     eval_split: test
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:     freq_mismatch: 0.0
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:     image_from: database
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:     import: recipe1m.datasets.factory
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:     interchange_ingrd_instr: None
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:     kw_path:
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:       test: /home/mshukor/data/recipe1m/clip_da/layer1_test_ingr_kw.json
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:       train: /home/mshukor/data/recipe1m/clip_da/layer1_train_ingr_kw.json
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:       val: /home/mshukor/data/recipe1m/clip_da/layer1_val_ingr_kw.json
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:     name: recipe1m
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:     nb_classes: 1048
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:     nb_threads: 4
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:     randkw_p: 0.3
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:     randkw_p_aux: 0.5
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:     remove_list: [ingredients,title]
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:     tokenized_raw_text: True
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:     train_split: None
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:     use_vcs: True
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:   engine:
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:     name: logger
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:     nb_epochs: 120
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:     print_freq: 10
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:     saving_criteria: ['eval_epoch.metric.med_im2recipe_mean:min', 'eval_epoch.metric.recall_at_1_im2recipe_mean:max']
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:   exp:
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:     checkpoint: None
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:     dir: /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:     resume: best_eval_epoch.metric.recall_at_1_im2recipe_mean
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:   misc:
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:     cuda: True
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:     data_parrallel: True
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:     logs_name: None
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:     overrite: False
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:     seed: 1338
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:   model:
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:     criterion:
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:         tri_cross_encoder: False
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:       itm_loss_weight: 1
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:       keep_background: False
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:       retrieval_strategy:
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:         aggregation: valid
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:         get_tokens: True
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:         margin: 0.05
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:         margin_params:
[S 2022-10-28 14:35:54] ...ytorch/bootstrap/run.py.75:           increment: 0.005
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:           increment_margin: True
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:           max_margin: 0.3
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:         name: triplet
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:         nb_samples: 9999
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:         sampling: max_negative
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:         substrategy: ['IRR', 'RII', 'SIRR', 'SRII']
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:         substrategy_weights: [1.0, 1.0, 0.1, 0.1]
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:     import: recipe1m.models.factory
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:     metric:
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       k_test: 10
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       nb_bags: 10
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       nb_matchs_per_bag: 1000
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       retrieval_dir: /tmp/recipe1m
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       save_ids: False
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       trijoint: True
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:     name: trijoint
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:     network:
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       activations: ['tanh', 'normalize']
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       aux_kwords: True
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       aux_kwords_encoder: True
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       avg_concat: True
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       bert_config: /home/mshukor/tfood/recipe1m/models/networks/recipe_networks/config_bert_albef.json
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       cat_pos: 0
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       checkpoint: None
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       class_attention: False
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       class_attention_fus: False
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       cls_token: False
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       cls_token_fus: False
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       cross_decoder_image: True
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       cross_decoder_recipe: True
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:         class_attention: False
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:         cls_token: False
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:         cross_attention: False
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:         cross_decoder: True
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:         double_decoder: False
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:         get_tokens: False
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:         load_backbones: False
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:         n_heads: 4
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:         n_layers: 4
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:         query_img: False
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       cross_transformer: False
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       dim_emb: 1024
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       dim_image_out: 2048
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       dim_ingr_out: 300
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       dim_instr_in: 1024
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       dim_instr_out: 1024
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       freeze_im: False
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       freeze_rec: False
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       get_tokens: True
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       get_tokens_cross_decoder_recipe: True
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       hidden_size: 512
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       image_backbone_name: clip
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       image_res: 224
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       kwords_same_level: False
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       last_cat_aux: True
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       n_heads: 4
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       n_heads_cross: 4
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       n_heads_cross_image: 4
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       n_heads_single: 4
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       n_layers: 2
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       n_layers_cross: 2
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       n_layers_cross_image: 1
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       n_layers_single: 2
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       num_hidden_layers_kw: 2
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       old: True
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       path_ingrs: /home/mshukor/data/recipe1m/text/vocab.pkl
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       path_vocab: /home/mshukor/data/recipe1m/text/vocab_all.txt
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       recipe_encoder: tfoodvicha-transformer
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       remove_additional_embedding: ['proj_image']
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       text_encoder: bert-base-uncased
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       vision_width: 768
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       vit_all_tokens: True
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       with_ingrs: True
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       with_instrs: True
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:       with_titles: True
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:     with_classif: False
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:   optimizer:
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:     batch_size_factor: None
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:     clip_grad: 8.0
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:     import: recipe1m.optimizers.factory
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:     lr: 1e-05
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:     lr_img: 1e-06
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:     name: trijoint_fixed_fine_tune
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:     switch_epoch: 20
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:   path_opts: /home/mshukor/tfood/recipe1m/options/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_noimgproj.yaml
[S 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.75:   view: ['logs:train_epoch.loss', 'logs:train_epoch.bad_pairs', 'logs:eval_epoch.metric.med_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_1_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_5_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_10_im2recipe_mean', 'logs:eval_epoch.metric.med_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_1_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_5_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_10_recipe2im_mean', 'logs:optimizer.is_optimizer_recipe&image', 'logs:optimizer.total_norm']
[I 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.76: posix.uname_result(sysname='Linux', nodename='DGX', release='5.4.0-124-generic', version='#140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022', machine='x86_64')
[I 2022-10-28 14:35:55] ...ytorch/bootstrap/run.py.80: Available GPUs: [0]
[I 2022-10-28 14:35:55] ...strap/engines/factory.py.9: Creating engine...
[I 2022-10-28 14:35:55] ...trap/datasets/factory.py.8: Creating dataset...
[I 2022-10-28 14:35:56] ...m/datasets/recipe1m.py.152: Load VCs...
[I 2022-10-28 14:35:57] ...m/datasets/recipe1m.py.165: randkw_p... None
[I 2022-10-28 14:37:00] ...m/datasets/recipe1m.py.310: recipe elements to remove: [ingredients,title]
[I 2022-10-28 14:37:00] ...rap/datasets/factory.py.21: Evaluation will take place on testset (51303 items)
[I 2022-10-28 14:37:00] ...strap/models/factory.py.21: Creating model...
[I 2022-10-28 14:37:01] ...e_networks/networks.py.203: Loading CLIP Visual model...
[I 2022-10-28 14:37:04] ...e_networks/networks.py.145: cuda
[I 2022-10-28 14:37:05] ...networks/tfood_vicha.py.49: Load VCs encoder...
[I 2022-10-28 14:37:08] ...els/metrics/trijoint.py.46: self.save_ids False
[I 2022-10-28 14:37:08] ...strap/models/factory.py.44: Enabling CUDA mode...
[I 2022-10-28 14:37:10] ...etworks/tfood_vicha.py.184: training aux kw modules
[I 2022-10-28 14:37:10] ...etworks/tfood_vicha.py.184: training aux kw modules
[I 2022-10-28 14:37:10] ...otstrap/views/factory.py.8: Creating views...
[I 2022-10-28 14:37:10] ...strap/engines/engine.py.98: Loading best_eval_epoch.metric.recall_at_1_im2recipe_mean checkpoint
[I 2022-10-28 14:37:10] ...trap/engines/engine.py.440: Loading model...
[I 2022-10-28 14:37:11] ...trap/engines/engine.py.453: Loading engine...
[I 2022-10-28 14:37:11] ...trap/engines/engine.py.108: Launching evaluation procedures
[I 2022-10-28 14:37:11] ...trap/engines/engine.py.268: Evaluating model on testset for epoch 116
[I 2022-10-28 14:37:11] ...m/datasets/recipe1m.py.400: Dataset will be sampled with "random" batch_sampler.
[I 2022-10-28 14:37:14] ...trap/engines/engine.py.321: eval: epoch 116 | batch 0/512
[I 2022-10-28 14:37:14] ...trap/engines/engine.py.325:       elapsed: 0:00:02 | left: 0:23:20
[I 2022-10-28 14:37:14] ...trap/engines/engine.py.326:       process: 2.73631 | load: 1.52173
[I 2022-10-28 14:37:17] ...trap/engines/engine.py.321: eval: epoch 116 | batch 10/512
[I 2022-10-28 14:37:17] ...trap/engines/engine.py.325:       elapsed: 0:00:05 | left: 0:04:55
[I 2022-10-28 14:37:17] ...trap/engines/engine.py.326:       process: 0.42705 | load: 0.00014
[I 2022-10-28 14:37:20] ...trap/engines/engine.py.321: eval: epoch 116 | batch 20/512
[I 2022-10-28 14:37:20] ...trap/engines/engine.py.325:       elapsed: 0:00:08 | left: 0:02:41
[I 2022-10-28 14:37:20] ...trap/engines/engine.py.326:       process: 0.29660 | load: 0.00014
[I 2022-10-28 14:37:23] ...trap/engines/engine.py.321: eval: epoch 116 | batch 30/512
[I 2022-10-28 14:37:23] ...trap/engines/engine.py.325:       elapsed: 0:00:11 | left: 0:02:25
[I 2022-10-28 14:37:23] ...trap/engines/engine.py.326:       process: 0.29161 | load: 0.00015
[I 2022-10-28 14:37:26] ...trap/engines/engine.py.321: eval: epoch 116 | batch 40/512
[I 2022-10-28 14:37:26] ...trap/engines/engine.py.325:       elapsed: 0:00:14 | left: 0:02:19
[I 2022-10-28 14:37:26] ...trap/engines/engine.py.326:       process: 0.29684 | load: 0.00015
[I 2022-10-28 14:37:29] ...trap/engines/engine.py.321: eval: epoch 116 | batch 50/512
[I 2022-10-28 14:37:29] ...trap/engines/engine.py.325:       elapsed: 0:00:17 | left: 0:02:18
[I 2022-10-28 14:37:29] ...trap/engines/engine.py.326:       process: 0.29553 | load: 0.00021
[I 2022-10-28 14:37:32] ...trap/engines/engine.py.321: eval: epoch 116 | batch 60/512
[I 2022-10-28 14:37:32] ...trap/engines/engine.py.325:       elapsed: 0:00:20 | left: 0:02:15
[I 2022-10-28 14:37:32] ...trap/engines/engine.py.326:       process: 0.29673 | load: 0.00019
[I 2022-10-28 14:37:35] ...trap/engines/engine.py.321: eval: epoch 116 | batch 70/512
[I 2022-10-28 14:37:35] ...trap/engines/engine.py.325:       elapsed: 0:00:23 | left: 0:02:12
[I 2022-10-28 14:37:35] ...trap/engines/engine.py.326:       process: 0.29466 | load: 0.00015
[I 2022-10-28 14:37:38] ...trap/engines/engine.py.321: eval: epoch 116 | batch 80/512
[I 2022-10-28 14:37:38] ...trap/engines/engine.py.325:       elapsed: 0:00:26 | left: 0:02:10
[I 2022-10-28 14:37:38] ...trap/engines/engine.py.326:       process: 0.29465 | load: 0.00021
[I 2022-10-28 14:37:41] ...trap/engines/engine.py.321: eval: epoch 116 | batch 90/512
[I 2022-10-28 14:37:41] ...trap/engines/engine.py.325:       elapsed: 0:00:29 | left: 0:02:05
[I 2022-10-28 14:37:41] ...trap/engines/engine.py.326:       process: 0.29493 | load: 0.00016
[I 2022-10-28 14:37:44] ...trap/engines/engine.py.321: eval: epoch 116 | batch 100/512
[I 2022-10-28 14:37:44] ...trap/engines/engine.py.325:       elapsed: 0:00:32 | left: 0:01:59
[I 2022-10-28 14:37:44] ...trap/engines/engine.py.326:       process: 0.25766 | load: 0.00019
[I 2022-10-28 14:37:47] ...trap/engines/engine.py.321: eval: epoch 116 | batch 110/512
[I 2022-10-28 14:37:47] ...trap/engines/engine.py.325:       elapsed: 0:00:35 | left: 0:01:44
[I 2022-10-28 14:37:47] ...trap/engines/engine.py.326:       process: 0.25559 | load: 0.00015
[I 2022-10-28 14:37:49] ...trap/engines/engine.py.321: eval: epoch 116 | batch 120/512
[I 2022-10-28 14:37:49] ...trap/engines/engine.py.325:       elapsed: 0:00:37 | left: 0:01:40
[I 2022-10-28 14:37:49] ...trap/engines/engine.py.326:       process: 0.25771 | load: 0.00013
[I 2022-10-28 14:37:52] ...trap/engines/engine.py.321: eval: epoch 116 | batch 130/512
[I 2022-10-28 14:37:52] ...trap/engines/engine.py.325:       elapsed: 0:00:40 | left: 0:01:39
[I 2022-10-28 14:37:52] ...trap/engines/engine.py.326:       process: 0.27462 | load: 0.00015
[I 2022-10-28 14:37:55] ...trap/engines/engine.py.321: eval: epoch 116 | batch 140/512
[I 2022-10-28 14:37:55] ...trap/engines/engine.py.325:       elapsed: 0:00:43 | left: 0:01:35
[I 2022-10-28 14:37:55] ...trap/engines/engine.py.326:       process: 0.25556 | load: 0.00013
[I 2022-10-28 14:37:57] ...trap/engines/engine.py.321: eval: epoch 116 | batch 150/512
[I 2022-10-28 14:37:57] ...trap/engines/engine.py.325:       elapsed: 0:00:45 | left: 0:01:33
[I 2022-10-28 14:37:57] ...trap/engines/engine.py.326:       process: 0.25409 | load: 0.00017
[I 2022-10-28 14:38:00] ...trap/engines/engine.py.321: eval: epoch 116 | batch 160/512
[I 2022-10-28 14:38:00] ...trap/engines/engine.py.325:       elapsed: 0:00:48 | left: 0:01:31
[I 2022-10-28 14:38:00] ...trap/engines/engine.py.326:       process: 0.25844 | load: 0.00013
[I 2022-10-28 14:38:02] ...trap/engines/engine.py.321: eval: epoch 116 | batch 170/512
[I 2022-10-28 14:38:02] ...trap/engines/engine.py.325:       elapsed: 0:00:50 | left: 0:01:29
[I 2022-10-28 14:38:02] ...trap/engines/engine.py.326:       process: 0.25475 | load: 0.00015
[I 2022-10-28 14:38:05] ...trap/engines/engine.py.321: eval: epoch 116 | batch 180/512
[I 2022-10-28 14:38:05] ...trap/engines/engine.py.325:       elapsed: 0:00:53 | left: 0:01:25
[I 2022-10-28 14:38:05] ...trap/engines/engine.py.326:       process: 0.25735 | load: 0.00014
[I 2022-10-28 14:38:08] ...trap/engines/engine.py.321: eval: epoch 116 | batch 190/512
[I 2022-10-28 14:38:08] ...trap/engines/engine.py.325:       elapsed: 0:00:56 | left: 0:01:22
[I 2022-10-28 14:38:08] ...trap/engines/engine.py.326:       process: 0.25634 | load: 0.00016
[I 2022-10-28 14:38:10] ...trap/engines/engine.py.321: eval: epoch 116 | batch 200/512
[I 2022-10-28 14:38:10] ...trap/engines/engine.py.325:       elapsed: 0:00:58 | left: 0:01:19
[I 2022-10-28 14:38:10] ...trap/engines/engine.py.326:       process: 0.25812 | load: 0.00013
[I 2022-10-28 14:38:13] ...trap/engines/engine.py.321: eval: epoch 116 | batch 210/512
[I 2022-10-28 14:38:13] ...trap/engines/engine.py.325:       elapsed: 0:01:01 | left: 0:01:17
[I 2022-10-28 14:38:13] ...trap/engines/engine.py.326:       process: 0.25164 | load: 0.00016
[I 2022-10-28 14:38:15] ...trap/engines/engine.py.321: eval: epoch 116 | batch 220/512
[I 2022-10-28 14:38:15] ...trap/engines/engine.py.325:       elapsed: 0:01:03 | left: 0:01:15
[I 2022-10-28 14:38:15] ...trap/engines/engine.py.326:       process: 0.25822 | load: 0.00015
[I 2022-10-28 14:38:18] ...trap/engines/engine.py.321: eval: epoch 116 | batch 230/512
[I 2022-10-28 14:38:18] ...trap/engines/engine.py.325:       elapsed: 0:01:06 | left: 0:01:11
[I 2022-10-28 14:38:18] ...trap/engines/engine.py.326:       process: 0.25180 | load: 0.00016
[I 2022-10-28 14:38:20] ...trap/engines/engine.py.321: eval: epoch 116 | batch 240/512
[I 2022-10-28 14:38:20] ...trap/engines/engine.py.325:       elapsed: 0:01:08 | left: 0:01:10
[I 2022-10-28 14:38:20] ...trap/engines/engine.py.326:       process: 0.25834 | load: 0.00018
[I 2022-10-28 14:38:23] ...trap/engines/engine.py.321: eval: epoch 116 | batch 250/512
[I 2022-10-28 14:38:23] ...trap/engines/engine.py.325:       elapsed: 0:01:11 | left: 0:01:07
[I 2022-10-28 14:38:23] ...trap/engines/engine.py.326:       process: 0.25417 | load: 0.00015
[I 2022-10-28 14:38:26] ...trap/engines/engine.py.321: eval: epoch 116 | batch 260/512
[I 2022-10-28 14:38:26] ...trap/engines/engine.py.325:       elapsed: 0:01:14 | left: 0:01:04
[I 2022-10-28 14:38:26] ...trap/engines/engine.py.326:       process: 0.25761 | load: 0.00014
[I 2022-10-28 14:38:28] ...trap/engines/engine.py.321: eval: epoch 116 | batch 270/512
[I 2022-10-28 14:38:28] ...trap/engines/engine.py.325:       elapsed: 0:01:16 | left: 0:01:02
[I 2022-10-28 14:38:28] ...trap/engines/engine.py.326:       process: 0.25511 | load: 0.00018
[I 2022-10-28 14:38:31] ...trap/engines/engine.py.321: eval: epoch 116 | batch 280/512
[I 2022-10-28 14:38:31] ...trap/engines/engine.py.325:       elapsed: 0:01:19 | left: 0:00:59
[I 2022-10-28 14:38:31] ...trap/engines/engine.py.326:       process: 0.25639 | load: 0.00017
[I 2022-10-28 14:38:33] ...trap/engines/engine.py.321: eval: epoch 116 | batch 290/512
[I 2022-10-28 14:38:33] ...trap/engines/engine.py.325:       elapsed: 0:01:21 | left: 0:00:56
[I 2022-10-28 14:38:33] ...trap/engines/engine.py.326:       process: 0.25664 | load: 0.00017
[I 2022-10-28 14:38:36] ...trap/engines/engine.py.321: eval: epoch 116 | batch 300/512
[I 2022-10-28 14:38:36] ...trap/engines/engine.py.325:       elapsed: 0:01:24 | left: 0:00:57
[I 2022-10-28 14:38:36] ...trap/engines/engine.py.326:       process: 0.31076 | load: 0.00017
[I 2022-10-28 14:38:38] ...trap/engines/engine.py.321: eval: epoch 116 | batch 310/512
[I 2022-10-28 14:38:38] ...trap/engines/engine.py.325:       elapsed: 0:01:26 | left: 0:00:52
[I 2022-10-28 14:38:38] ...trap/engines/engine.py.326:       process: 0.25507 | load: 0.00017
[I 2022-10-28 14:38:41] ...trap/engines/engine.py.321: eval: epoch 116 | batch 320/512
[I 2022-10-28 14:38:41] ...trap/engines/engine.py.325:       elapsed: 0:01:29 | left: 0:00:49
[I 2022-10-28 14:38:41] ...trap/engines/engine.py.326:       process: 0.25737 | load: 0.00015
[I 2022-10-28 14:38:44] ...trap/engines/engine.py.321: eval: epoch 116 | batch 330/512
[I 2022-10-28 14:38:44] ...trap/engines/engine.py.325:       elapsed: 0:01:32 | left: 0:00:46
[I 2022-10-28 14:38:44] ...trap/engines/engine.py.326:       process: 0.25756 | load: 0.00016
[I 2022-10-28 14:38:46] ...trap/engines/engine.py.321: eval: epoch 116 | batch 340/512
[I 2022-10-28 14:38:46] ...trap/engines/engine.py.325:       elapsed: 0:01:34 | left: 0:00:44
[I 2022-10-28 14:38:46] ...trap/engines/engine.py.326:       process: 0.26050 | load: 0.00014
[I 2022-10-28 14:38:49] ...trap/engines/engine.py.321: eval: epoch 116 | batch 350/512
[I 2022-10-28 14:38:49] ...trap/engines/engine.py.325:       elapsed: 0:01:37 | left: 0:00:41
[I 2022-10-28 14:38:49] ...trap/engines/engine.py.326:       process: 0.25629 | load: 0.00015
[I 2022-10-28 14:38:51] ...trap/engines/engine.py.321: eval: epoch 116 | batch 360/512
[I 2022-10-28 14:38:51] ...trap/engines/engine.py.325:       elapsed: 0:01:39 | left: 0:00:38
[I 2022-10-28 14:38:51] ...trap/engines/engine.py.326:       process: 0.25635 | load: 0.00014
[I 2022-10-28 14:38:54] ...trap/engines/engine.py.321: eval: epoch 116 | batch 370/512
[I 2022-10-28 14:38:54] ...trap/engines/engine.py.325:       elapsed: 0:01:42 | left: 0:00:36
[I 2022-10-28 14:38:54] ...trap/engines/engine.py.326:       process: 0.25686 | load: 0.00016
[I 2022-10-28 14:38:56] ...trap/engines/engine.py.321: eval: epoch 116 | batch 380/512
[I 2022-10-28 14:38:56] ...trap/engines/engine.py.325:       elapsed: 0:01:44 | left: 0:00:33
[I 2022-10-28 14:38:56] ...trap/engines/engine.py.326:       process: 0.25817 | load: 0.00015
[I 2022-10-28 14:38:59] ...trap/engines/engine.py.321: eval: epoch 116 | batch 390/512
[I 2022-10-28 14:38:59] ...trap/engines/engine.py.325:       elapsed: 0:01:47 | left: 0:00:31
[I 2022-10-28 14:38:59] ...trap/engines/engine.py.326:       process: 0.25812 | load: 0.00015
[I 2022-10-28 14:39:02] ...trap/engines/engine.py.321: eval: epoch 116 | batch 400/512
[I 2022-10-28 14:39:02] ...trap/engines/engine.py.325:       elapsed: 0:01:50 | left: 0:00:28
[I 2022-10-28 14:39:02] ...trap/engines/engine.py.326:       process: 0.25705 | load: 0.00016
[I 2022-10-28 14:39:04] ...trap/engines/engine.py.321: eval: epoch 116 | batch 410/512
[I 2022-10-28 14:39:04] ...trap/engines/engine.py.325:       elapsed: 0:01:52 | left: 0:00:26
[I 2022-10-28 14:39:04] ...trap/engines/engine.py.326:       process: 0.25792 | load: 0.00017
[I 2022-10-28 14:39:07] ...trap/engines/engine.py.321: eval: epoch 116 | batch 420/512
[I 2022-10-28 14:39:07] ...trap/engines/engine.py.325:       elapsed: 0:01:55 | left: 0:00:23
[I 2022-10-28 14:39:07] ...trap/engines/engine.py.326:       process: 0.25700 | load: 0.00017
[I 2022-10-28 14:39:09] ...trap/engines/engine.py.321: eval: epoch 116 | batch 430/512
[I 2022-10-28 14:39:09] ...trap/engines/engine.py.325:       elapsed: 0:01:57 | left: 0:00:21
[I 2022-10-28 14:39:09] ...trap/engines/engine.py.326:       process: 0.25560 | load: 0.00015
[I 2022-10-28 14:39:12] ...trap/engines/engine.py.321: eval: epoch 116 | batch 440/512
[I 2022-10-28 14:39:12] ...trap/engines/engine.py.325:       elapsed: 0:02:00 | left: 0:00:18
[I 2022-10-28 14:39:12] ...trap/engines/engine.py.326:       process: 0.25615 | load: 0.00017
[I 2022-10-28 14:39:14] ...trap/engines/engine.py.321: eval: epoch 116 | batch 450/512
[I 2022-10-28 14:39:14] ...trap/engines/engine.py.325:       elapsed: 0:02:02 | left: 0:00:15
[I 2022-10-28 14:39:14] ...trap/engines/engine.py.326:       process: 0.25294 | load: 0.00015
[I 2022-10-28 14:39:17] ...trap/engines/engine.py.321: eval: epoch 116 | batch 460/512
[I 2022-10-28 14:39:17] ...trap/engines/engine.py.325:       elapsed: 0:02:05 | left: 0:00:13
[I 2022-10-28 14:39:17] ...trap/engines/engine.py.326:       process: 0.25543 | load: 0.00017
[I 2022-10-28 14:39:19] ...trap/engines/engine.py.321: eval: epoch 116 | batch 470/512
[I 2022-10-28 14:39:19] ...trap/engines/engine.py.325:       elapsed: 0:02:08 | left: 0:00:10
[I 2022-10-28 14:39:19] ...trap/engines/engine.py.326:       process: 0.25602 | load: 0.00017
[I 2022-10-28 14:39:22] ...trap/engines/engine.py.321: eval: epoch 116 | batch 480/512
[I 2022-10-28 14:39:22] ...trap/engines/engine.py.325:       elapsed: 0:02:10 | left: 0:00:08
[I 2022-10-28 14:39:22] ...trap/engines/engine.py.326:       process: 0.25732 | load: 0.00017
[I 2022-10-28 14:39:25] ...trap/engines/engine.py.321: eval: epoch 116 | batch 490/512
[I 2022-10-28 14:39:25] ...trap/engines/engine.py.325:       elapsed: 0:02:13 | left: 0:00:05
[I 2022-10-28 14:39:25] ...trap/engines/engine.py.326:       process: 0.25598 | load: 0.00015
[I 2022-10-28 14:39:27] ...trap/engines/engine.py.321: eval: epoch 116 | batch 500/512
[I 2022-10-28 14:39:27] ...trap/engines/engine.py.325:       elapsed: 0:02:15 | left: 0:00:03
[I 2022-10-28 14:39:27] ...trap/engines/engine.py.326:       process: 0.25695 | load: 0.00019
[I 2022-10-28 14:39:30] ...trap/engines/engine.py.321: eval: epoch 116 | batch 510/512
[I 2022-10-28 14:39:30] ...trap/engines/engine.py.325:       elapsed: 0:02:18 | left: 0:00:00
[I 2022-10-28 14:39:30] ...trap/engines/engine.py.326:       process: 0.25130 | load: 0.00011
[S 2022-10-28 14:39:31] ...trap/engines/engine.py.113: eval_epoch.epoch: 116
[I 2022-10-28 14:39:31] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs
[S 2022-10-28 14:39:35] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 37.90
[S 2022-10-28 14:39:35] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 10.10
[S 2022-10-28 14:39:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.072700
[S 2022-10-28 14:39:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.204000
[S 2022-10-28 14:39:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.290200
[S 2022-10-28 14:39:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.177100
[S 2022-10-28 14:39:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.401300
[S 2022-10-28 14:39:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.509700
[S 2022-10-28 14:39:35] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 3.87
[S 2022-10-28 14:39:35] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 1.30
[S 2022-10-28 14:39:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.008331
[S 2022-10-28 14:39:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.009920
[S 2022-10-28 14:39:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.014407
[S 2022-10-28 14:39:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.004949
[S 2022-10-28 14:39:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.016211
[S 2022-10-28 14:39:35] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.022392
[I 2022-10-28 14:39:35] ...trap/engines/engine.py.115: Ending evaluation procedures
[W 2022-10-28 14:39:36] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json'
[W 2022-10-28 14:39:36] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[W 2022-10-28 14:39:36] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[I 2022-10-28 14:39:40] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html
[I 2022-10-28 16:33:49] ...ytorch/bootstrap/run.py.73: Saving environment info
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75: options: 
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:   dataset:
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     aux_kw_path:
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       test: /home/mshukor/data/recipe1m/clip_da/layer1_test_titles_kw.json
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       train: /home/mshukor/data/recipe1m/clip_da/layer1_train_titles_kw.json
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       val: /home/mshukor/data/recipe1m/clip_da/layer1_val_titles_kw.json
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     batch_sampler: triplet_classif
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     batch_size: 100
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     database: lmdb
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     dir: /home/mshukor/data/recipe1m
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     eval_split: test
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     freq_mismatch: 0.0
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     image_from: database
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     import: recipe1m.datasets.factory
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     interchange_ingrd_instr: None
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     kw_path:
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       test: /home/mshukor/data/recipe1m/clip_da/layer1_test_ingr_kw.json
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       train: /home/mshukor/data/recipe1m/clip_da/layer1_train_ingr_kw.json
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       val: /home/mshukor/data/recipe1m/clip_da/layer1_val_ingr_kw.json
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     name: recipe1m
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     nb_classes: 1048
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     nb_threads: 4
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     randkw_p: 0.3
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     randkw_p_aux: 0.5
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     remove_list: None
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     tokenized_raw_text: True
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     train_split: None
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     use_vcs: True
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:   engine:
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     name: logger
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     nb_epochs: 120
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     print_freq: 10
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     saving_criteria: ['eval_epoch.metric.med_im2recipe_mean:min', 'eval_epoch.metric.recall_at_1_im2recipe_mean:max']
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:   exp:
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     checkpoint: None
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     dir: /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     resume: best_eval_epoch.metric.recall_at_1_im2recipe_mean
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:   misc:
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     cuda: True
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     data_parrallel: True
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     logs_name: None
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     overrite: False
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     seed: 1338
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:   model:
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     criterion:
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:         tri_cross_encoder: False
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       itm_loss_weight: 1
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       keep_background: False
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       retrieval_strategy:
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:         aggregation: valid
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:         get_tokens: True
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:         margin: 0.05
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:         margin_params:
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:           increment: 0.005
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:           increment_margin: True
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:           max_margin: 0.3
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:         name: triplet
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:         nb_samples: 9999
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:         sampling: max_negative
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:         substrategy: ['IRR', 'RII', 'SIRR', 'SRII']
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:         substrategy_weights: [1.0, 1.0, 0.1, 0.1]
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     import: recipe1m.models.factory
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     metric:
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       k_test: 10
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       nb_bags: 2
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       nb_matchs_per_bag: 20000
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       retrieval_dir: /tmp/recipe1m
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       save_ids: False
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       trijoint: True
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     name: trijoint
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     network:
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       activations: ['tanh', 'normalize']
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       aux_kwords: True
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       aux_kwords_encoder: True
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       avg_concat: True
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       bert_config: /home/mshukor/tfood/recipe1m/models/networks/recipe_networks/config_bert_albef.json
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       cat_pos: 0
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       checkpoint: None
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       class_attention: False
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       class_attention_fus: False
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       cls_token: False
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       cls_token_fus: False
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       cross_decoder_image: True
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       cross_decoder_recipe: True
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:         class_attention: False
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:         cls_token: False
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:         cross_attention: False
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:         cross_decoder: True
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:         double_decoder: False
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:         get_tokens: False
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:         load_backbones: False
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:         n_heads: 4
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:         n_layers: 4
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:         query_img: False
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       cross_transformer: False
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       dim_emb: 1024
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       dim_image_out: 2048
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       dim_ingr_out: 300
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       dim_instr_in: 1024
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       dim_instr_out: 1024
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       freeze_im: False
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       freeze_rec: False
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       get_tokens: True
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       get_tokens_cross_decoder_recipe: True
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       hidden_size: 512
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       image_backbone_name: clip
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       image_res: 224
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       kwords_same_level: False
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       last_cat_aux: True
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       n_heads: 4
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       n_heads_cross: 4
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       n_heads_cross_image: 4
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       n_heads_single: 4
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       n_layers: 2
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       n_layers_cross: 2
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       n_layers_cross_image: 1
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       n_layers_single: 2
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       num_hidden_layers_kw: 2
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       old: True
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       path_ingrs: /home/mshukor/data/recipe1m/text/vocab.pkl
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       path_vocab: /home/mshukor/data/recipe1m/text/vocab_all.txt
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       recipe_encoder: tfoodvicha-transformer
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       remove_additional_embedding: ['proj_image']
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       text_encoder: bert-base-uncased
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       vision_width: 768
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       vit_all_tokens: True
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       with_ingrs: True
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       with_instrs: True
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:       with_titles: True
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     with_classif: False
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:   optimizer:
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     batch_size_factor: None
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     clip_grad: 8.0
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     import: recipe1m.optimizers.factory
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     lr: 1e-05
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     lr_img: 1e-06
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     name: trijoint_fixed_fine_tune
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:     switch_epoch: 20
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:   path_opts: /home/mshukor/tfood/recipe1m/options/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_noimgproj.yaml
[S 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.75:   view: ['logs:train_epoch.loss', 'logs:train_epoch.bad_pairs', 'logs:eval_epoch.metric.med_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_1_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_5_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_10_im2recipe_mean', 'logs:eval_epoch.metric.med_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_1_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_5_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_10_recipe2im_mean', 'logs:optimizer.is_optimizer_recipe&image', 'logs:optimizer.total_norm']
[I 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.76: posix.uname_result(sysname='Linux', nodename='DGX', release='5.4.0-124-generic', version='#140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022', machine='x86_64')
[I 2022-10-28 16:33:50] ...ytorch/bootstrap/run.py.80: Available GPUs: [0]
[I 2022-10-28 16:33:50] ...strap/engines/factory.py.9: Creating engine...
[I 2022-10-28 16:33:50] ...trap/datasets/factory.py.8: Creating dataset...
[I 2022-10-28 16:33:52] ...m/datasets/recipe1m.py.152: Load VCs...
[I 2022-10-28 16:33:52] ...m/datasets/recipe1m.py.165: randkw_p... None
[I 2022-10-28 16:34:55] ...m/datasets/recipe1m.py.310: recipe elements to remove: None
[I 2022-10-28 16:34:55] ...rap/datasets/factory.py.21: Evaluation will take place on testset (51303 items)
[I 2022-10-28 16:34:55] ...strap/models/factory.py.21: Creating model...
[I 2022-10-28 16:34:56] ...e_networks/networks.py.203: Loading CLIP Visual model...
[I 2022-10-28 16:34:59] ...e_networks/networks.py.145: cuda
[I 2022-10-28 16:35:01] ...networks/tfood_vicha.py.49: Load VCs encoder...
[I 2022-10-28 16:35:04] ...els/metrics/trijoint.py.46: self.save_ids False
[I 2022-10-28 16:35:04] ...strap/models/factory.py.44: Enabling CUDA mode...
[I 2022-10-28 16:35:06] ...etworks/tfood_vicha.py.184: training aux kw modules
[I 2022-10-28 16:35:06] ...etworks/tfood_vicha.py.184: training aux kw modules
[I 2022-10-28 16:35:06] ...otstrap/views/factory.py.8: Creating views...
[I 2022-10-28 16:35:06] ...strap/engines/engine.py.98: Loading best_eval_epoch.metric.recall_at_1_im2recipe_mean checkpoint
[I 2022-10-28 16:35:06] ...trap/engines/engine.py.440: Loading model...
[I 2022-10-28 16:35:07] ...trap/engines/engine.py.453: Loading engine...
[I 2022-10-28 16:35:07] ...trap/engines/engine.py.108: Launching evaluation procedures
[I 2022-10-28 16:35:07] ...trap/engines/engine.py.268: Evaluating model on testset for epoch 116
[I 2022-10-28 16:35:07] ...m/datasets/recipe1m.py.400: Dataset will be sampled with "random" batch_sampler.
[I 2022-10-28 16:35:10] ...trap/engines/engine.py.321: eval: epoch 116 | batch 0/512
[I 2022-10-28 16:35:10] ...trap/engines/engine.py.325:       elapsed: 0:00:02 | left: 0:23:50
[I 2022-10-28 16:35:10] ...trap/engines/engine.py.326:       process: 2.79454 | load: 1.50132
[I 2022-10-28 16:35:13] ...trap/engines/engine.py.321: eval: epoch 116 | batch 10/512
[I 2022-10-28 16:35:13] ...trap/engines/engine.py.325:       elapsed: 0:00:06 | left: 0:04:59
[I 2022-10-28 16:35:13] ...trap/engines/engine.py.326:       process: 0.32252 | load: 0.00016
[I 2022-10-28 16:35:16] ...trap/engines/engine.py.321: eval: epoch 116 | batch 20/512
[I 2022-10-28 16:35:16] ...trap/engines/engine.py.325:       elapsed: 0:00:09 | left: 0:02:54
[I 2022-10-28 16:35:16] ...trap/engines/engine.py.326:       process: 0.32086 | load: 0.00015
[I 2022-10-28 16:35:20] ...trap/engines/engine.py.321: eval: epoch 116 | batch 30/512
[I 2022-10-28 16:35:20] ...trap/engines/engine.py.325:       elapsed: 0:00:12 | left: 0:02:39
[I 2022-10-28 16:35:20] ...trap/engines/engine.py.326:       process: 0.32910 | load: 0.00016
[I 2022-10-28 16:35:23] ...trap/engines/engine.py.321: eval: epoch 116 | batch 40/512
[I 2022-10-28 16:35:23] ...trap/engines/engine.py.325:       elapsed: 0:00:15 | left: 0:02:35
[I 2022-10-28 16:35:23] ...trap/engines/engine.py.326:       process: 0.32844 | load: 0.00016
[I 2022-10-28 16:35:26] ...trap/engines/engine.py.321: eval: epoch 116 | batch 50/512
[I 2022-10-28 16:35:26] ...trap/engines/engine.py.325:       elapsed: 0:00:19 | left: 0:02:31
[I 2022-10-28 16:35:26] ...trap/engines/engine.py.326:       process: 0.32772 | load: 0.00017
[I 2022-10-28 16:35:29] ...trap/engines/engine.py.321: eval: epoch 116 | batch 60/512
[I 2022-10-28 16:35:29] ...trap/engines/engine.py.325:       elapsed: 0:00:22 | left: 0:02:32
[I 2022-10-28 16:35:29] ...trap/engines/engine.py.326:       process: 0.36104 | load: 0.00015
[I 2022-10-28 16:35:33] ...trap/engines/engine.py.321: eval: epoch 116 | batch 70/512
[I 2022-10-28 16:35:33] ...trap/engines/engine.py.325:       elapsed: 0:00:25 | left: 0:02:25
[I 2022-10-28 16:35:33] ...trap/engines/engine.py.326:       process: 0.33047 | load: 0.00018
[I 2022-10-28 16:35:36] ...trap/engines/engine.py.321: eval: epoch 116 | batch 80/512
[I 2022-10-28 16:35:36] ...trap/engines/engine.py.325:       elapsed: 0:00:29 | left: 0:02:22
[I 2022-10-28 16:35:36] ...trap/engines/engine.py.326:       process: 0.32231 | load: 0.00017
[I 2022-10-28 16:35:39] ...trap/engines/engine.py.321: eval: epoch 116 | batch 90/512
[I 2022-10-28 16:35:39] ...trap/engines/engine.py.325:       elapsed: 0:00:32 | left: 0:02:21
[I 2022-10-28 16:35:39] ...trap/engines/engine.py.326:       process: 0.33804 | load: 0.00016
[I 2022-10-28 16:35:43] ...trap/engines/engine.py.321: eval: epoch 116 | batch 100/512
[I 2022-10-28 16:35:43] ...trap/engines/engine.py.325:       elapsed: 0:00:35 | left: 0:02:16
[I 2022-10-28 16:35:43] ...trap/engines/engine.py.326:       process: 0.33557 | load: 0.00016
[I 2022-10-28 16:35:46] ...trap/engines/engine.py.321: eval: epoch 116 | batch 110/512
[I 2022-10-28 16:35:46] ...trap/engines/engine.py.325:       elapsed: 0:00:39 | left: 0:02:14
[I 2022-10-28 16:35:46] ...trap/engines/engine.py.326:       process: 0.34561 | load: 0.00017
[I 2022-10-28 16:35:50] ...trap/engines/engine.py.321: eval: epoch 116 | batch 120/512
[I 2022-10-28 16:35:50] ...trap/engines/engine.py.325:       elapsed: 0:00:42 | left: 0:02:12
[I 2022-10-28 16:35:50] ...trap/engines/engine.py.326:       process: 0.33499 | load: 0.00016
[I 2022-10-28 16:35:53] ...trap/engines/engine.py.321: eval: epoch 116 | batch 130/512
[I 2022-10-28 16:35:53] ...trap/engines/engine.py.325:       elapsed: 0:00:45 | left: 0:02:06
[I 2022-10-28 16:35:53] ...trap/engines/engine.py.326:       process: 0.32621 | load: 0.00016
[I 2022-10-28 16:35:56] ...trap/engines/engine.py.321: eval: epoch 116 | batch 140/512
[I 2022-10-28 16:35:56] ...trap/engines/engine.py.325:       elapsed: 0:00:49 | left: 0:02:03
[I 2022-10-28 16:35:56] ...trap/engines/engine.py.326:       process: 0.32285 | load: 0.00017
[I 2022-10-28 16:36:00] ...trap/engines/engine.py.321: eval: epoch 116 | batch 150/512
[I 2022-10-28 16:36:00] ...trap/engines/engine.py.325:       elapsed: 0:00:52 | left: 0:02:01
[I 2022-10-28 16:36:00] ...trap/engines/engine.py.326:       process: 0.32559 | load: 0.00017
[I 2022-10-28 16:36:03] ...trap/engines/engine.py.321: eval: epoch 116 | batch 160/512
[I 2022-10-28 16:36:03] ...trap/engines/engine.py.325:       elapsed: 0:00:55 | left: 0:01:56
[I 2022-10-28 16:36:03] ...trap/engines/engine.py.326:       process: 0.33601 | load: 0.00018
[I 2022-10-28 16:36:06] ...trap/engines/engine.py.321: eval: epoch 116 | batch 170/512
[I 2022-10-28 16:36:06] ...trap/engines/engine.py.325:       elapsed: 0:00:59 | left: 0:01:52
[I 2022-10-28 16:36:06] ...trap/engines/engine.py.326:       process: 0.33535 | load: 0.00019
[I 2022-10-28 16:36:09] ...trap/engines/engine.py.321: eval: epoch 116 | batch 180/512
[I 2022-10-28 16:36:09] ...trap/engines/engine.py.325:       elapsed: 0:01:02 | left: 0:01:51
[I 2022-10-28 16:36:09] ...trap/engines/engine.py.326:       process: 0.32563 | load: 0.00016
[I 2022-10-28 16:36:13] ...trap/engines/engine.py.321: eval: epoch 116 | batch 190/512
[I 2022-10-28 16:36:13] ...trap/engines/engine.py.325:       elapsed: 0:01:05 | left: 0:01:48
[I 2022-10-28 16:36:13] ...trap/engines/engine.py.326:       process: 0.33394 | load: 0.00015
[I 2022-10-28 16:36:16] ...trap/engines/engine.py.321: eval: epoch 116 | batch 200/512
[I 2022-10-28 16:36:16] ...trap/engines/engine.py.325:       elapsed: 0:01:09 | left: 0:01:44
[I 2022-10-28 16:36:16] ...trap/engines/engine.py.326:       process: 0.33312 | load: 0.00019
[I 2022-10-28 16:36:20] ...trap/engines/engine.py.321: eval: epoch 116 | batch 210/512
[I 2022-10-28 16:36:20] ...trap/engines/engine.py.325:       elapsed: 0:01:12 | left: 0:01:41
[I 2022-10-28 16:36:20] ...trap/engines/engine.py.326:       process: 0.33235 | load: 0.00016
[I 2022-10-28 16:36:23] ...trap/engines/engine.py.321: eval: epoch 116 | batch 220/512
[I 2022-10-28 16:36:23] ...trap/engines/engine.py.325:       elapsed: 0:01:15 | left: 0:01:36
[I 2022-10-28 16:36:23] ...trap/engines/engine.py.326:       process: 0.33382 | load: 0.00016
[I 2022-10-28 16:36:26] ...trap/engines/engine.py.321: eval: epoch 116 | batch 230/512
[I 2022-10-28 16:36:26] ...trap/engines/engine.py.325:       elapsed: 0:01:19 | left: 0:01:33
[I 2022-10-28 16:36:26] ...trap/engines/engine.py.326:       process: 0.33179 | load: 0.00017
[I 2022-10-28 16:36:30] ...trap/engines/engine.py.321: eval: epoch 116 | batch 240/512
[I 2022-10-28 16:36:30] ...trap/engines/engine.py.325:       elapsed: 0:01:22 | left: 0:01:31
[I 2022-10-28 16:36:30] ...trap/engines/engine.py.326:       process: 0.33087 | load: 0.00016
[I 2022-10-28 16:36:33] ...trap/engines/engine.py.321: eval: epoch 116 | batch 250/512
[I 2022-10-28 16:36:33] ...trap/engines/engine.py.325:       elapsed: 0:01:25 | left: 0:01:27
[I 2022-10-28 16:36:33] ...trap/engines/engine.py.326:       process: 0.33018 | load: 0.00016
[I 2022-10-28 16:36:36] ...trap/engines/engine.py.321: eval: epoch 116 | batch 260/512
[I 2022-10-28 16:36:36] ...trap/engines/engine.py.325:       elapsed: 0:01:29 | left: 0:01:23
[I 2022-10-28 16:36:36] ...trap/engines/engine.py.326:       process: 0.33391 | load: 0.00019
[I 2022-10-28 16:36:40] ...trap/engines/engine.py.321: eval: epoch 116 | batch 270/512
[I 2022-10-28 16:36:40] ...trap/engines/engine.py.325:       elapsed: 0:01:32 | left: 0:01:21
[I 2022-10-28 16:36:40] ...trap/engines/engine.py.326:       process: 0.34805 | load: 0.00016
[I 2022-10-28 16:36:43] ...trap/engines/engine.py.321: eval: epoch 116 | batch 280/512
[I 2022-10-28 16:36:43] ...trap/engines/engine.py.325:       elapsed: 0:01:35 | left: 0:01:17
[I 2022-10-28 16:36:43] ...trap/engines/engine.py.326:       process: 0.33040 | load: 0.00019
[I 2022-10-28 16:36:46] ...trap/engines/engine.py.321: eval: epoch 116 | batch 290/512
[I 2022-10-28 16:36:46] ...trap/engines/engine.py.325:       elapsed: 0:01:39 | left: 0:01:13
[I 2022-10-28 16:36:46] ...trap/engines/engine.py.326:       process: 0.32884 | load: 0.00020
[I 2022-10-28 16:36:50] ...trap/engines/engine.py.321: eval: epoch 116 | batch 300/512
[I 2022-10-28 16:36:50] ...trap/engines/engine.py.325:       elapsed: 0:01:42 | left: 0:01:11
[I 2022-10-28 16:36:50] ...trap/engines/engine.py.326:       process: 0.32934 | load: 0.00018
[I 2022-10-28 16:36:53] ...trap/engines/engine.py.321: eval: epoch 116 | batch 310/512
[I 2022-10-28 16:36:53] ...trap/engines/engine.py.325:       elapsed: 0:01:45 | left: 0:01:07
[I 2022-10-28 16:36:53] ...trap/engines/engine.py.326:       process: 0.33319 | load: 0.00017
[I 2022-10-28 16:36:56] ...trap/engines/engine.py.321: eval: epoch 116 | batch 320/512
[I 2022-10-28 16:36:56] ...trap/engines/engine.py.325:       elapsed: 0:01:49 | left: 0:01:03
[I 2022-10-28 16:36:56] ...trap/engines/engine.py.326:       process: 0.32140 | load: 0.00017
[I 2022-10-28 16:37:00] ...trap/engines/engine.py.321: eval: epoch 116 | batch 330/512
[I 2022-10-28 16:37:00] ...trap/engines/engine.py.325:       elapsed: 0:01:52 | left: 0:01:00
[I 2022-10-28 16:37:00] ...trap/engines/engine.py.326:       process: 0.33019 | load: 0.00017
[I 2022-10-28 16:37:03] ...trap/engines/engine.py.321: eval: epoch 116 | batch 340/512
[I 2022-10-28 16:37:03] ...trap/engines/engine.py.325:       elapsed: 0:01:55 | left: 0:00:56
[I 2022-10-28 16:37:03] ...trap/engines/engine.py.326:       process: 0.33254 | load: 0.00019
[I 2022-10-28 16:37:06] ...trap/engines/engine.py.321: eval: epoch 116 | batch 350/512
[I 2022-10-28 16:37:06] ...trap/engines/engine.py.325:       elapsed: 0:01:59 | left: 0:00:53
[I 2022-10-28 16:37:06] ...trap/engines/engine.py.326:       process: 0.32347 | load: 0.00017
[I 2022-10-28 16:37:10] ...trap/engines/engine.py.321: eval: epoch 116 | batch 360/512
[I 2022-10-28 16:37:10] ...trap/engines/engine.py.325:       elapsed: 0:02:02 | left: 0:00:51
[I 2022-10-28 16:37:10] ...trap/engines/engine.py.326:       process: 0.36230 | load: 0.00019
[I 2022-10-28 16:37:13] ...trap/engines/engine.py.321: eval: epoch 116 | batch 370/512
[I 2022-10-28 16:37:13] ...trap/engines/engine.py.325:       elapsed: 0:02:05 | left: 0:00:47
[I 2022-10-28 16:37:13] ...trap/engines/engine.py.326:       process: 0.33007 | load: 0.00016
[I 2022-10-28 16:37:16] ...trap/engines/engine.py.321: eval: epoch 116 | batch 380/512
[I 2022-10-28 16:37:16] ...trap/engines/engine.py.325:       elapsed: 0:02:09 | left: 0:00:43
[I 2022-10-28 16:37:16] ...trap/engines/engine.py.326:       process: 0.33268 | load: 0.00017
[I 2022-10-28 16:37:20] ...trap/engines/engine.py.321: eval: epoch 116 | batch 390/512
[I 2022-10-28 16:37:20] ...trap/engines/engine.py.325:       elapsed: 0:02:12 | left: 0:00:42
[I 2022-10-28 16:37:20] ...trap/engines/engine.py.326:       process: 0.43240 | load: 0.00017
[I 2022-10-28 16:37:23] ...trap/engines/engine.py.321: eval: epoch 116 | batch 400/512
[I 2022-10-28 16:37:23] ...trap/engines/engine.py.325:       elapsed: 0:02:15 | left: 0:00:36
[I 2022-10-28 16:37:23] ...trap/engines/engine.py.326:       process: 0.29674 | load: 0.00017
[I 2022-10-28 16:37:26] ...trap/engines/engine.py.321: eval: epoch 116 | batch 410/512
[I 2022-10-28 16:37:26] ...trap/engines/engine.py.325:       elapsed: 0:02:18 | left: 0:00:30
[I 2022-10-28 16:37:26] ...trap/engines/engine.py.326:       process: 0.28817 | load: 0.00017
[I 2022-10-28 16:37:29] ...trap/engines/engine.py.321: eval: epoch 116 | batch 420/512
[I 2022-10-28 16:37:29] ...trap/engines/engine.py.325:       elapsed: 0:02:21 | left: 0:00:26
[I 2022-10-28 16:37:29] ...trap/engines/engine.py.326:       process: 0.28664 | load: 0.00018
[I 2022-10-28 16:37:32] ...trap/engines/engine.py.321: eval: epoch 116 | batch 430/512
[I 2022-10-28 16:37:32] ...trap/engines/engine.py.325:       elapsed: 0:02:24 | left: 0:00:24
[I 2022-10-28 16:37:32] ...trap/engines/engine.py.326:       process: 0.29092 | load: 0.00015
[I 2022-10-28 16:37:34] ...trap/engines/engine.py.321: eval: epoch 116 | batch 440/512
[I 2022-10-28 16:37:35] ...trap/engines/engine.py.325:       elapsed: 0:02:27 | left: 0:00:20
[I 2022-10-28 16:37:35] ...trap/engines/engine.py.326:       process: 0.28640 | load: 0.00017
[I 2022-10-28 16:37:37] ...trap/engines/engine.py.321: eval: epoch 116 | batch 450/512
[I 2022-10-28 16:37:37] ...trap/engines/engine.py.325:       elapsed: 0:02:30 | left: 0:00:17
[I 2022-10-28 16:37:37] ...trap/engines/engine.py.326:       process: 0.28118 | load: 0.00018
[I 2022-10-28 16:37:40] ...trap/engines/engine.py.321: eval: epoch 116 | batch 460/512
[I 2022-10-28 16:37:40] ...trap/engines/engine.py.325:       elapsed: 0:02:33 | left: 0:00:14
[I 2022-10-28 16:37:40] ...trap/engines/engine.py.326:       process: 0.28972 | load: 0.00018
[I 2022-10-28 16:37:43] ...trap/engines/engine.py.321: eval: epoch 116 | batch 470/512
[I 2022-10-28 16:37:43] ...trap/engines/engine.py.325:       elapsed: 0:02:36 | left: 0:00:11
[I 2022-10-28 16:37:43] ...trap/engines/engine.py.326:       process: 0.28445 | load: 0.00015
[I 2022-10-28 16:37:46] ...trap/engines/engine.py.321: eval: epoch 116 | batch 480/512
[I 2022-10-28 16:37:46] ...trap/engines/engine.py.325:       elapsed: 0:02:39 | left: 0:00:09
[I 2022-10-28 16:37:46] ...trap/engines/engine.py.326:       process: 0.28560 | load: 0.00016
[I 2022-10-28 16:37:49] ...trap/engines/engine.py.321: eval: epoch 116 | batch 490/512
[I 2022-10-28 16:37:49] ...trap/engines/engine.py.325:       elapsed: 0:02:41 | left: 0:00:06
[I 2022-10-28 16:37:49] ...trap/engines/engine.py.326:       process: 0.27989 | load: 0.00018
[I 2022-10-28 16:37:52] ...trap/engines/engine.py.321: eval: epoch 116 | batch 500/512
[I 2022-10-28 16:37:52] ...trap/engines/engine.py.325:       elapsed: 0:02:44 | left: 0:00:03
[I 2022-10-28 16:37:52] ...trap/engines/engine.py.326:       process: 0.28572 | load: 0.00018
[I 2022-10-28 16:37:55] ...trap/engines/engine.py.321: eval: epoch 116 | batch 510/512
[I 2022-10-28 16:37:55] ...trap/engines/engine.py.325:       elapsed: 0:02:47 | left: 0:00:00
[I 2022-10-28 16:37:55] ...trap/engines/engine.py.326:       process: 0.28107 | load: 0.00009
[S 2022-10-28 16:37:56] ...trap/engines/engine.py.113: eval_epoch.epoch: 116
[I 2022-10-28 16:37:56] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 2 x 20000 matchs
[S 2022-10-28 16:41:07] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 2.50
[S 2022-10-28 16:41:07] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 3.00
[S 2022-10-28 16:41:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.379625
[S 2022-10-28 16:41:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.644850
[S 2022-10-28 16:41:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.739525
[S 2022-10-28 16:41:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.365100
[S 2022-10-28 16:41:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.635475
[S 2022-10-28 16:41:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.734200
[S 2022-10-28 16:41:07] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.500000
[S 2022-10-28 16:41:07] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000
[S 2022-10-28 16:41:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.003325
[S 2022-10-28 16:41:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.004900
[S 2022-10-28 16:41:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.002625
[S 2022-10-28 16:41:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.005450
[S 2022-10-28 16:41:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.000575
[S 2022-10-28 16:41:07] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.002100
[I 2022-10-28 16:41:08] ...trap/engines/engine.py.115: Ending evaluation procedures
[W 2022-10-28 16:41:09] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json'
[W 2022-10-28 16:41:09] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[W 2022-10-28 16:41:09] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[I 2022-10-28 16:41:13] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html
[I 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.73: Saving environment info
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75: options: 
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:   dataset:
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     aux_kw_path:
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       test: /home/mshukor/data/recipe1m/clip_da/layer1_test_titles_kw.json
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       train: /home/mshukor/data/recipe1m/clip_da/layer1_train_titles_kw.json
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       val: /home/mshukor/data/recipe1m/clip_da/layer1_val_titles_kw.json
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     batch_sampler: triplet_classif
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     batch_size: 100
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     database: lmdb
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     dir: /home/mshukor/data/recipe1m
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     eval_split: test
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     freq_mismatch: 0.0
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     image_from: database
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     import: recipe1m.datasets.factory
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     interchange_ingrd_instr: None
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     kw_path:
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       test: /home/mshukor/data/recipe1m/clip_da/layer1_test_ingr_kw.json
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       train: /home/mshukor/data/recipe1m/clip_da/layer1_train_ingr_kw.json
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       val: /home/mshukor/data/recipe1m/clip_da/layer1_val_ingr_kw.json
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     name: recipe1m
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     nb_classes: 1048
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     nb_threads: 4
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     randkw_p: 0.3
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     randkw_p_aux: 0.5
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     remove_list: None
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     tokenized_raw_text: True
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     train_split: None
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     use_vcs: True
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:   engine:
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     name: logger
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     nb_epochs: 120
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     print_freq: 10
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     saving_criteria: ['eval_epoch.metric.med_im2recipe_mean:min', 'eval_epoch.metric.recall_at_1_im2recipe_mean:max']
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:   exp:
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     checkpoint: None
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     dir: /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     resume: best_eval_epoch.metric.recall_at_1_im2recipe_mean
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:   misc:
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     cuda: True
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     data_parrallel: True
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     logs_name: None
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     overrite: False
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     seed: 1338
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:   model:
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     criterion:
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:         tri_cross_encoder: False
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       itm_loss_weight: 1
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       keep_background: False
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       retrieval_strategy:
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:         aggregation: valid
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:         get_tokens: True
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:         margin: 0.05
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:         margin_params:
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:           increment: 0.005
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:           increment_margin: True
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:           max_margin: 0.3
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:         name: triplet
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:         nb_samples: 9999
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:         sampling: max_negative
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:         substrategy: ['IRR', 'RII', 'SIRR', 'SRII']
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:         substrategy_weights: [1.0, 1.0, 0.1, 0.1]
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     import: recipe1m.models.factory
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     metric:
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       k_test: 10
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       nb_bags: 1
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       nb_matchs_per_bag: 30000
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       retrieval_dir: /tmp/recipe1m
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       save_ids: False
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       trijoint: True
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     name: trijoint
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     network:
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       activations: ['tanh', 'normalize']
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       aux_kwords: True
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       aux_kwords_encoder: True
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       avg_concat: True
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       bert_config: /home/mshukor/tfood/recipe1m/models/networks/recipe_networks/config_bert_albef.json
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       cat_pos: 0
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       checkpoint: None
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       class_attention: False
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       class_attention_fus: False
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       cls_token: False
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       cls_token_fus: False
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       cross_decoder_image: True
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       cross_decoder_recipe: True
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:         class_attention: False
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:         cls_token: False
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:         cross_attention: False
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:         cross_decoder: True
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:         double_decoder: False
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:         get_tokens: False
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:         load_backbones: False
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:         n_heads: 4
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:         n_layers: 4
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:         query_img: False
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       cross_transformer: False
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       dim_emb: 1024
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       dim_image_out: 2048
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       dim_ingr_out: 300
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       dim_instr_in: 1024
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       dim_instr_out: 1024
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       freeze_im: False
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       freeze_rec: False
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       get_tokens: True
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       get_tokens_cross_decoder_recipe: True
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       hidden_size: 512
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       image_backbone_name: clip
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       image_res: 224
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       kwords_same_level: False
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       last_cat_aux: True
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       n_heads: 4
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       n_heads_cross: 4
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       n_heads_cross_image: 4
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       n_heads_single: 4
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       n_layers: 2
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       n_layers_cross: 2
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       n_layers_cross_image: 1
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       n_layers_single: 2
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       num_hidden_layers_kw: 2
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       old: True
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       path_ingrs: /home/mshukor/data/recipe1m/text/vocab.pkl
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       path_vocab: /home/mshukor/data/recipe1m/text/vocab_all.txt
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       recipe_encoder: tfoodvicha-transformer
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       remove_additional_embedding: ['proj_image']
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       text_encoder: bert-base-uncased
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       vision_width: 768
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       vit_all_tokens: True
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       with_ingrs: True
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       with_instrs: True
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:       with_titles: True
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     with_classif: False
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:   optimizer:
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     batch_size_factor: None
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     clip_grad: 8.0
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     import: recipe1m.optimizers.factory
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     lr: 1e-05
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     lr_img: 1e-06
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     name: trijoint_fixed_fine_tune
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:     switch_epoch: 20
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:   path_opts: /home/mshukor/tfood/recipe1m/options/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_noimgproj.yaml
[S 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.75:   view: ['logs:train_epoch.loss', 'logs:train_epoch.bad_pairs', 'logs:eval_epoch.metric.med_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_1_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_5_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_10_im2recipe_mean', 'logs:eval_epoch.metric.med_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_1_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_5_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_10_recipe2im_mean', 'logs:optimizer.is_optimizer_recipe&image', 'logs:optimizer.total_norm']
[I 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.76: posix.uname_result(sysname='Linux', nodename='DGX', release='5.4.0-124-generic', version='#140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022', machine='x86_64')
[I 2022-10-28 16:43:29] ...ytorch/bootstrap/run.py.80: Available GPUs: [0]
[I 2022-10-28 16:43:29] ...strap/engines/factory.py.9: Creating engine...
[I 2022-10-28 16:43:29] ...trap/datasets/factory.py.8: Creating dataset...
[I 2022-10-28 16:43:31] ...m/datasets/recipe1m.py.152: Load VCs...
[I 2022-10-28 16:43:32] ...m/datasets/recipe1m.py.165: randkw_p... None
[I 2022-10-28 16:44:34] ...m/datasets/recipe1m.py.310: recipe elements to remove: None
[I 2022-10-28 16:44:34] ...rap/datasets/factory.py.21: Evaluation will take place on testset (51303 items)
[I 2022-10-28 16:44:34] ...strap/models/factory.py.21: Creating model...
[I 2022-10-28 16:44:35] ...e_networks/networks.py.203: Loading CLIP Visual model...
[I 2022-10-28 16:44:38] ...e_networks/networks.py.145: cuda
[I 2022-10-28 16:44:40] ...networks/tfood_vicha.py.49: Load VCs encoder...
[I 2022-10-28 16:44:43] ...els/metrics/trijoint.py.46: self.save_ids False
[I 2022-10-28 16:44:43] ...strap/models/factory.py.44: Enabling CUDA mode...
[I 2022-10-28 16:44:45] ...etworks/tfood_vicha.py.184: training aux kw modules
[I 2022-10-28 16:44:45] ...etworks/tfood_vicha.py.184: training aux kw modules
[I 2022-10-28 16:44:45] ...otstrap/views/factory.py.8: Creating views...
[I 2022-10-28 16:44:45] ...strap/engines/engine.py.98: Loading best_eval_epoch.metric.recall_at_1_im2recipe_mean checkpoint
[I 2022-10-28 16:44:45] ...trap/engines/engine.py.440: Loading model...
[I 2022-10-28 16:44:46] ...trap/engines/engine.py.453: Loading engine...
[I 2022-10-28 16:44:46] ...trap/engines/engine.py.108: Launching evaluation procedures
[I 2022-10-28 16:44:46] ...trap/engines/engine.py.268: Evaluating model on testset for epoch 116
[I 2022-10-28 16:44:46] ...m/datasets/recipe1m.py.400: Dataset will be sampled with "random" batch_sampler.
[I 2022-10-28 16:44:49] ...trap/engines/engine.py.321: eval: epoch 116 | batch 0/512
[I 2022-10-28 16:44:49] ...trap/engines/engine.py.325:       elapsed: 0:00:02 | left: 0:23:48
[I 2022-10-28 16:44:49] ...trap/engines/engine.py.326:       process: 2.78977 | load: 1.50533
[I 2022-10-28 16:44:52] ...trap/engines/engine.py.321: eval: epoch 116 | batch 10/512
[I 2022-10-28 16:44:52] ...trap/engines/engine.py.325:       elapsed: 0:00:06 | left: 0:04:57
[I 2022-10-28 16:44:52] ...trap/engines/engine.py.326:       process: 0.32216 | load: 0.00016
[I 2022-10-28 16:44:55] ...trap/engines/engine.py.321: eval: epoch 116 | batch 20/512
[I 2022-10-28 16:44:55] ...trap/engines/engine.py.325:       elapsed: 0:00:09 | left: 0:02:55
[I 2022-10-28 16:44:55] ...trap/engines/engine.py.326:       process: 0.32726 | load: 0.00015
[I 2022-10-28 16:44:58] ...trap/engines/engine.py.321: eval: epoch 116 | batch 30/512
[I 2022-10-28 16:44:58] ...trap/engines/engine.py.325:       elapsed: 0:00:12 | left: 0:02:38
[I 2022-10-28 16:44:58] ...trap/engines/engine.py.326:       process: 0.32444 | load: 0.00015
[I 2022-10-28 16:45:02] ...trap/engines/engine.py.321: eval: epoch 116 | batch 40/512
[I 2022-10-28 16:45:02] ...trap/engines/engine.py.325:       elapsed: 0:00:15 | left: 0:02:34
[I 2022-10-28 16:45:02] ...trap/engines/engine.py.326:       process: 0.32533 | load: 0.00015
[I 2022-10-28 16:45:05] ...trap/engines/engine.py.321: eval: epoch 116 | batch 50/512
[I 2022-10-28 16:45:05] ...trap/engines/engine.py.325:       elapsed: 0:00:19 | left: 0:02:31
[I 2022-10-28 16:45:05] ...trap/engines/engine.py.326:       process: 0.32739 | load: 0.00017
[I 2022-10-28 16:45:08] ...trap/engines/engine.py.321: eval: epoch 116 | batch 60/512
[I 2022-10-28 16:45:08] ...trap/engines/engine.py.325:       elapsed: 0:00:22 | left: 0:02:33
[I 2022-10-28 16:45:08] ...trap/engines/engine.py.326:       process: 0.37321 | load: 0.00016
[I 2022-10-28 16:45:12] ...trap/engines/engine.py.321: eval: epoch 116 | batch 70/512
[I 2022-10-28 16:45:12] ...trap/engines/engine.py.325:       elapsed: 0:00:25 | left: 0:02:29
[I 2022-10-28 16:45:12] ...trap/engines/engine.py.326:       process: 0.33256 | load: 0.00015
[I 2022-10-28 16:45:15] ...trap/engines/engine.py.321: eval: epoch 116 | batch 80/512
[I 2022-10-28 16:45:15] ...trap/engines/engine.py.325:       elapsed: 0:00:29 | left: 0:02:22
[I 2022-10-28 16:45:15] ...trap/engines/engine.py.326:       process: 0.32438 | load: 0.00018
[I 2022-10-28 16:45:18] ...trap/engines/engine.py.321: eval: epoch 116 | batch 90/512
[I 2022-10-28 16:45:18] ...trap/engines/engine.py.325:       elapsed: 0:00:32 | left: 0:02:20
[I 2022-10-28 16:45:18] ...trap/engines/engine.py.326:       process: 0.33722 | load: 0.00019
[I 2022-10-28 16:45:22] ...trap/engines/engine.py.321: eval: epoch 116 | batch 100/512
[I 2022-10-28 16:45:22] ...trap/engines/engine.py.325:       elapsed: 0:00:35 | left: 0:02:18
[I 2022-10-28 16:45:22] ...trap/engines/engine.py.326:       process: 0.33621 | load: 0.00017
[I 2022-10-28 16:45:25] ...trap/engines/engine.py.321: eval: epoch 116 | batch 110/512
[I 2022-10-28 16:45:25] ...trap/engines/engine.py.325:       elapsed: 0:00:39 | left: 0:02:13
[I 2022-10-28 16:45:25] ...trap/engines/engine.py.326:       process: 0.32877 | load: 0.00018
[I 2022-10-28 16:45:28] ...trap/engines/engine.py.321: eval: epoch 116 | batch 120/512
[I 2022-10-28 16:45:28] ...trap/engines/engine.py.325:       elapsed: 0:00:42 | left: 0:02:11
[I 2022-10-28 16:45:28] ...trap/engines/engine.py.326:       process: 0.33312 | load: 0.00016
[I 2022-10-28 16:45:32] ...trap/engines/engine.py.321: eval: epoch 116 | batch 130/512
[I 2022-10-28 16:45:32] ...trap/engines/engine.py.325:       elapsed: 0:00:45 | left: 0:02:08
[I 2022-10-28 16:45:32] ...trap/engines/engine.py.326:       process: 0.33655 | load: 0.00020
[I 2022-10-28 16:45:35] ...trap/engines/engine.py.321: eval: epoch 116 | batch 140/512
[I 2022-10-28 16:45:35] ...trap/engines/engine.py.325:       elapsed: 0:00:49 | left: 0:02:03
[I 2022-10-28 16:45:35] ...trap/engines/engine.py.326:       process: 0.32897 | load: 0.00016
[I 2022-10-28 16:45:38] ...trap/engines/engine.py.321: eval: epoch 116 | batch 150/512
[I 2022-10-28 16:45:38] ...trap/engines/engine.py.325:       elapsed: 0:00:52 | left: 0:02:00
[I 2022-10-28 16:45:38] ...trap/engines/engine.py.326:       process: 0.33204 | load: 0.00017
[I 2022-10-28 16:45:42] ...trap/engines/engine.py.321: eval: epoch 116 | batch 160/512
[I 2022-10-28 16:45:42] ...trap/engines/engine.py.325:       elapsed: 0:00:55 | left: 0:01:58
[I 2022-10-28 16:45:42] ...trap/engines/engine.py.326:       process: 0.33249 | load: 0.00018
[I 2022-10-28 16:45:45] ...trap/engines/engine.py.321: eval: epoch 116 | batch 170/512
[I 2022-10-28 16:45:45] ...trap/engines/engine.py.325:       elapsed: 0:00:59 | left: 0:01:52
[I 2022-10-28 16:45:45] ...trap/engines/engine.py.326:       process: 0.33229 | load: 0.00017
[I 2022-10-28 16:45:48] ...trap/engines/engine.py.321: eval: epoch 116 | batch 180/512
[I 2022-10-28 16:45:48] ...trap/engines/engine.py.325:       elapsed: 0:01:02 | left: 0:01:50
[I 2022-10-28 16:45:48] ...trap/engines/engine.py.326:       process: 0.32622 | load: 0.00017
[I 2022-10-28 16:45:52] ...trap/engines/engine.py.321: eval: epoch 116 | batch 190/512
[I 2022-10-28 16:45:52] ...trap/engines/engine.py.325:       elapsed: 0:01:06 | left: 0:01:50
[I 2022-10-28 16:45:52] ...trap/engines/engine.py.326:       process: 0.35201 | load: 0.00016
[I 2022-10-28 16:45:55] ...trap/engines/engine.py.321: eval: epoch 116 | batch 200/512
[I 2022-10-28 16:45:55] ...trap/engines/engine.py.325:       elapsed: 0:01:09 | left: 0:01:44
[I 2022-10-28 16:45:55] ...trap/engines/engine.py.326:       process: 0.33445 | load: 0.00018
[I 2022-10-28 16:45:59] ...trap/engines/engine.py.321: eval: epoch 116 | batch 210/512
[I 2022-10-28 16:45:59] ...trap/engines/engine.py.325:       elapsed: 0:01:12 | left: 0:01:40
[I 2022-10-28 16:45:59] ...trap/engines/engine.py.326:       process: 0.32996 | load: 0.00016
[I 2022-10-28 16:46:02] ...trap/engines/engine.py.321: eval: epoch 116 | batch 220/512
[I 2022-10-28 16:46:02] ...trap/engines/engine.py.325:       elapsed: 0:01:16 | left: 0:01:38
[I 2022-10-28 16:46:02] ...trap/engines/engine.py.326:       process: 0.33619 | load: 0.00016
[I 2022-10-28 16:46:05] ...trap/engines/engine.py.321: eval: epoch 116 | batch 230/512
[I 2022-10-28 16:46:05] ...trap/engines/engine.py.325:       elapsed: 0:01:19 | left: 0:01:34
[I 2022-10-28 16:46:05] ...trap/engines/engine.py.326:       process: 0.33415 | load: 0.00015
[I 2022-10-28 16:46:09] ...trap/engines/engine.py.321: eval: epoch 116 | batch 240/512
[I 2022-10-28 16:46:09] ...trap/engines/engine.py.325:       elapsed: 0:01:22 | left: 0:01:30
[I 2022-10-28 16:46:09] ...trap/engines/engine.py.326:       process: 0.33245 | load: 0.00016
[I 2022-10-28 16:46:12] ...trap/engines/engine.py.321: eval: epoch 116 | batch 250/512
[I 2022-10-28 16:46:12] ...trap/engines/engine.py.325:       elapsed: 0:01:26 | left: 0:01:27
[I 2022-10-28 16:46:12] ...trap/engines/engine.py.326:       process: 0.33551 | load: 0.00022
[I 2022-10-28 16:46:15] ...trap/engines/engine.py.321: eval: epoch 116 | batch 260/512
[I 2022-10-28 16:46:15] ...trap/engines/engine.py.325:       elapsed: 0:01:29 | left: 0:01:23
[I 2022-10-28 16:46:15] ...trap/engines/engine.py.326:       process: 0.33373 | load: 0.00017
[I 2022-10-28 16:46:19] ...trap/engines/engine.py.321: eval: epoch 116 | batch 270/512
[I 2022-10-28 16:46:19] ...trap/engines/engine.py.325:       elapsed: 0:01:32 | left: 0:01:23
[I 2022-10-28 16:46:19] ...trap/engines/engine.py.326:       process: 0.33479 | load: 0.00016
[I 2022-10-28 16:46:22] ...trap/engines/engine.py.321: eval: epoch 116 | batch 280/512
[I 2022-10-28 16:46:22] ...trap/engines/engine.py.325:       elapsed: 0:01:36 | left: 0:01:18
[I 2022-10-28 16:46:22] ...trap/engines/engine.py.326:       process: 0.33393 | load: 0.00020
[I 2022-10-28 16:46:25] ...trap/engines/engine.py.321: eval: epoch 116 | batch 290/512
[I 2022-10-28 16:46:25] ...trap/engines/engine.py.325:       elapsed: 0:01:39 | left: 0:01:13
[I 2022-10-28 16:46:25] ...trap/engines/engine.py.326:       process: 0.32279 | load: 0.00019
[I 2022-10-28 16:46:29] ...trap/engines/engine.py.321: eval: epoch 116 | batch 300/512
[I 2022-10-28 16:46:29] ...trap/engines/engine.py.325:       elapsed: 0:01:42 | left: 0:01:08
[I 2022-10-28 16:46:29] ...trap/engines/engine.py.326:       process: 0.28168 | load: 0.00017
[I 2022-10-28 16:46:32] ...trap/engines/engine.py.321: eval: epoch 116 | batch 310/512
[I 2022-10-28 16:46:32] ...trap/engines/engine.py.325:       elapsed: 0:01:45 | left: 0:00:59
[I 2022-10-28 16:46:32] ...trap/engines/engine.py.326:       process: 0.29006 | load: 0.00018
[I 2022-10-28 16:46:34] ...trap/engines/engine.py.321: eval: epoch 116 | batch 320/512
[I 2022-10-28 16:46:34] ...trap/engines/engine.py.325:       elapsed: 0:01:48 | left: 0:00:54
[I 2022-10-28 16:46:34] ...trap/engines/engine.py.326:       process: 0.28157 | load: 0.00014
[I 2022-10-28 16:46:37] ...trap/engines/engine.py.321: eval: epoch 116 | batch 330/512
[I 2022-10-28 16:46:37] ...trap/engines/engine.py.325:       elapsed: 0:01:51 | left: 0:00:51
[I 2022-10-28 16:46:37] ...trap/engines/engine.py.326:       process: 0.28218 | load: 0.00017
[I 2022-10-28 16:46:40] ...trap/engines/engine.py.321: eval: epoch 116 | batch 340/512
[I 2022-10-28 16:46:40] ...trap/engines/engine.py.325:       elapsed: 0:01:54 | left: 0:00:49
[I 2022-10-28 16:46:40] ...trap/engines/engine.py.326:       process: 0.28816 | load: 0.00017
[I 2022-10-28 16:46:43] ...trap/engines/engine.py.321: eval: epoch 116 | batch 350/512
[I 2022-10-28 16:46:43] ...trap/engines/engine.py.325:       elapsed: 0:01:57 | left: 0:00:46
[I 2022-10-28 16:46:43] ...trap/engines/engine.py.326:       process: 0.28673 | load: 0.00022
[I 2022-10-28 16:46:46] ...trap/engines/engine.py.321: eval: epoch 116 | batch 360/512
[I 2022-10-28 16:46:46] ...trap/engines/engine.py.325:       elapsed: 0:02:00 | left: 0:00:43
[I 2022-10-28 16:46:46] ...trap/engines/engine.py.326:       process: 0.28619 | load: 0.00015
[I 2022-10-28 16:46:49] ...trap/engines/engine.py.321: eval: epoch 116 | batch 370/512
[I 2022-10-28 16:46:49] ...trap/engines/engine.py.325:       elapsed: 0:02:02 | left: 0:00:40
[I 2022-10-28 16:46:49] ...trap/engines/engine.py.326:       process: 0.28805 | load: 0.00022
[I 2022-10-28 16:46:52] ...trap/engines/engine.py.321: eval: epoch 116 | batch 380/512
[I 2022-10-28 16:46:52] ...trap/engines/engine.py.325:       elapsed: 0:02:05 | left: 0:00:37
[I 2022-10-28 16:46:52] ...trap/engines/engine.py.326:       process: 0.28634 | load: 0.00015
[I 2022-10-28 16:46:54] ...trap/engines/engine.py.321: eval: epoch 116 | batch 390/512
[I 2022-10-28 16:46:54] ...trap/engines/engine.py.325:       elapsed: 0:02:08 | left: 0:00:34
[I 2022-10-28 16:46:54] ...trap/engines/engine.py.326:       process: 0.28795 | load: 0.00019
[I 2022-10-28 16:46:57] ...trap/engines/engine.py.321: eval: epoch 116 | batch 400/512
[I 2022-10-28 16:46:57] ...trap/engines/engine.py.325:       elapsed: 0:02:11 | left: 0:00:31
[I 2022-10-28 16:46:57] ...trap/engines/engine.py.326:       process: 0.28269 | load: 0.00017
[I 2022-10-28 16:47:00] ...trap/engines/engine.py.321: eval: epoch 116 | batch 410/512
[I 2022-10-28 16:47:00] ...trap/engines/engine.py.325:       elapsed: 0:02:14 | left: 0:00:29
[I 2022-10-28 16:47:00] ...trap/engines/engine.py.326:       process: 0.29081 | load: 0.00020
[I 2022-10-28 16:47:03] ...trap/engines/engine.py.321: eval: epoch 116 | batch 420/512
[I 2022-10-28 16:47:03] ...trap/engines/engine.py.325:       elapsed: 0:02:17 | left: 0:00:26
[I 2022-10-28 16:47:03] ...trap/engines/engine.py.326:       process: 0.28540 | load: 0.00015
[I 2022-10-28 16:47:06] ...trap/engines/engine.py.321: eval: epoch 116 | batch 430/512
[I 2022-10-28 16:47:06] ...trap/engines/engine.py.325:       elapsed: 0:02:20 | left: 0:00:23
[I 2022-10-28 16:47:06] ...trap/engines/engine.py.326:       process: 0.28693 | load: 0.00022
[I 2022-10-28 16:47:09] ...trap/engines/engine.py.321: eval: epoch 116 | batch 440/512
[I 2022-10-28 16:47:09] ...trap/engines/engine.py.325:       elapsed: 0:02:22 | left: 0:00:20
[I 2022-10-28 16:47:09] ...trap/engines/engine.py.326:       process: 0.28492 | load: 0.00015
[I 2022-10-28 16:47:12] ...trap/engines/engine.py.321: eval: epoch 116 | batch 450/512
[I 2022-10-28 16:47:12] ...trap/engines/engine.py.325:       elapsed: 0:02:25 | left: 0:00:17
[I 2022-10-28 16:47:12] ...trap/engines/engine.py.326:       process: 0.28457 | load: 0.00022
[I 2022-10-28 16:47:14] ...trap/engines/engine.py.321: eval: epoch 116 | batch 460/512
[I 2022-10-28 16:47:14] ...trap/engines/engine.py.325:       elapsed: 0:02:28 | left: 0:00:14
[I 2022-10-28 16:47:14] ...trap/engines/engine.py.326:       process: 0.28651 | load: 0.00016
[I 2022-10-28 16:47:17] ...trap/engines/engine.py.321: eval: epoch 116 | batch 470/512
[I 2022-10-28 16:47:17] ...trap/engines/engine.py.325:       elapsed: 0:02:31 | left: 0:00:12
[I 2022-10-28 16:47:17] ...trap/engines/engine.py.326:       process: 0.28553 | load: 0.00022
[I 2022-10-28 16:47:20] ...trap/engines/engine.py.321: eval: epoch 116 | batch 480/512
[I 2022-10-28 16:47:20] ...trap/engines/engine.py.325:       elapsed: 0:02:34 | left: 0:00:09
[I 2022-10-28 16:47:20] ...trap/engines/engine.py.326:       process: 0.28503 | load: 0.00015
[I 2022-10-28 16:47:23] ...trap/engines/engine.py.321: eval: epoch 116 | batch 490/512
[I 2022-10-28 16:47:23] ...trap/engines/engine.py.325:       elapsed: 0:02:37 | left: 0:00:06
[I 2022-10-28 16:47:23] ...trap/engines/engine.py.326:       process: 0.28143 | load: 0.00022
[I 2022-10-28 16:47:26] ...trap/engines/engine.py.321: eval: epoch 116 | batch 500/512
[I 2022-10-28 16:47:26] ...trap/engines/engine.py.325:       elapsed: 0:02:40 | left: 0:00:03
[I 2022-10-28 16:47:26] ...trap/engines/engine.py.326:       process: 0.28503 | load: 0.00015
[I 2022-10-28 16:47:29] ...trap/engines/engine.py.321: eval: epoch 116 | batch 510/512
[I 2022-10-28 16:47:29] ...trap/engines/engine.py.325:       elapsed: 0:02:43 | left: 0:00:00
[I 2022-10-28 16:47:29] ...trap/engines/engine.py.326:       process: 0.28222 | load: 0.00012
[S 2022-10-28 16:47:30] ...trap/engines/engine.py.113: eval_epoch.epoch: 116
[I 2022-10-28 16:47:30] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 1 x 30000 matchs
[S 2022-10-28 16:51:05] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 3.00
[S 2022-10-28 16:51:05] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 4.00
[S 2022-10-28 16:51:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.334967
[S 2022-10-28 16:51:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.589567
[S 2022-10-28 16:51:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.691800
[S 2022-10-28 16:51:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.321000
[S 2022-10-28 16:51:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.578233
[S 2022-10-28 16:51:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.682133
[S 2022-10-28 16:51:05] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000
[S 2022-10-28 16:51:05] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000
[S 2022-10-28 16:51:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.000000
[S 2022-10-28 16:51:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.000000
[S 2022-10-28 16:51:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.000000
[S 2022-10-28 16:51:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.000000
[S 2022-10-28 16:51:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.000000
[S 2022-10-28 16:51:05] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.000000
[I 2022-10-28 16:51:07] ...trap/engines/engine.py.115: Ending evaluation procedures
[W 2022-10-28 16:51:07] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json'
[W 2022-10-28 16:51:07] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[W 2022-10-28 16:51:07] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[I 2022-10-28 16:51:11] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html
[I 2022-10-28 16:53:26] ...ytorch/bootstrap/run.py.73: Saving environment info
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75: options: 
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:   dataset:
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     aux_kw_path:
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       test: /home/mshukor/data/recipe1m/clip_da/layer1_test_titles_kw.json
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       train: /home/mshukor/data/recipe1m/clip_da/layer1_train_titles_kw.json
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       val: /home/mshukor/data/recipe1m/clip_da/layer1_val_titles_kw.json
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     batch_sampler: triplet_classif
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     batch_size: 100
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     database: lmdb
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     dir: /home/mshukor/data/recipe1m
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     eval_split: test
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     freq_mismatch: 0.0
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     image_from: database
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     import: recipe1m.datasets.factory
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     interchange_ingrd_instr: None
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     kw_path:
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       test: /home/mshukor/data/recipe1m/clip_da/layer1_test_ingr_kw.json
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       train: /home/mshukor/data/recipe1m/clip_da/layer1_train_ingr_kw.json
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       val: /home/mshukor/data/recipe1m/clip_da/layer1_val_ingr_kw.json
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     name: recipe1m
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     nb_classes: 1048
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     nb_threads: 4
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     randkw_p: 0.3
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     randkw_p_aux: 0.5
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     remove_list: None
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     tokenized_raw_text: True
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     train_split: None
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     use_vcs: True
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:   engine:
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     name: logger
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     nb_epochs: 120
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     print_freq: 10
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     saving_criteria: ['eval_epoch.metric.med_im2recipe_mean:min', 'eval_epoch.metric.recall_at_1_im2recipe_mean:max']
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:   exp:
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     checkpoint: None
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     dir: /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     resume: best_eval_epoch.metric.recall_at_1_im2recipe_mean
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:   misc:
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     cuda: True
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     data_parrallel: True
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     logs_name: None
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     overrite: False
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     seed: 1338
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:   model:
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     criterion:
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:         tri_cross_encoder: False
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       itm_loss_weight: 1
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       keep_background: False
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       retrieval_strategy:
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:         aggregation: valid
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:         get_tokens: True
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:         margin: 0.05
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:         margin_params:
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:           increment: 0.005
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:           increment_margin: True
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:           max_margin: 0.3
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:         name: triplet
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:         nb_samples: 9999
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:         sampling: max_negative
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:         substrategy: ['IRR', 'RII', 'SIRR', 'SRII']
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:         substrategy_weights: [1.0, 1.0, 0.1, 0.1]
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     import: recipe1m.models.factory
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     metric:
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       k_test: 10
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       nb_bags: 1
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       nb_matchs_per_bag: 40000
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       retrieval_dir: /tmp/recipe1m
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       save_ids: False
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       trijoint: True
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     name: trijoint
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     network:
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       activations: ['tanh', 'normalize']
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       aux_kwords: True
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       aux_kwords_encoder: True
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       avg_concat: True
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       bert_config: /home/mshukor/tfood/recipe1m/models/networks/recipe_networks/config_bert_albef.json
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       cat_pos: 0
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       checkpoint: None
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       class_attention: False
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       class_attention_fus: False
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       cls_token: False
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       cls_token_fus: False
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       cross_decoder_image: True
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       cross_decoder_recipe: True
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:         class_attention: False
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:         cls_token: False
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:         cross_attention: False
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:         cross_decoder: True
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:         double_decoder: False
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:         get_tokens: False
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:         load_backbones: False
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:         n_heads: 4
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:         n_layers: 4
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:         query_img: False
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       cross_transformer: False
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       dim_emb: 1024
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       dim_image_out: 2048
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       dim_ingr_out: 300
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       dim_instr_in: 1024
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       dim_instr_out: 1024
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       freeze_im: False
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       freeze_rec: False
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       get_tokens: True
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       get_tokens_cross_decoder_recipe: True
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       hidden_size: 512
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       image_backbone_name: clip
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       image_res: 224
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       kwords_same_level: False
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       last_cat_aux: True
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       n_heads: 4
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       n_heads_cross: 4
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       n_heads_cross_image: 4
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       n_heads_single: 4
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       n_layers: 2
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       n_layers_cross: 2
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       n_layers_cross_image: 1
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       n_layers_single: 2
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       num_hidden_layers_kw: 2
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       old: True
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       path_ingrs: /home/mshukor/data/recipe1m/text/vocab.pkl
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       path_vocab: /home/mshukor/data/recipe1m/text/vocab_all.txt
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       recipe_encoder: tfoodvicha-transformer
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       remove_additional_embedding: ['proj_image']
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       text_encoder: bert-base-uncased
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       vision_width: 768
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       vit_all_tokens: True
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       with_ingrs: True
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       with_instrs: True
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:       with_titles: True
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     with_classif: False
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:   optimizer:
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     batch_size_factor: None
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     clip_grad: 8.0
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     import: recipe1m.optimizers.factory
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     lr: 1e-05
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     lr_img: 1e-06
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     name: trijoint_fixed_fine_tune
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:     switch_epoch: 20
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:   path_opts: /home/mshukor/tfood/recipe1m/options/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_noimgproj.yaml
[S 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.75:   view: ['logs:train_epoch.loss', 'logs:train_epoch.bad_pairs', 'logs:eval_epoch.metric.med_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_1_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_5_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_10_im2recipe_mean', 'logs:eval_epoch.metric.med_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_1_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_5_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_10_recipe2im_mean', 'logs:optimizer.is_optimizer_recipe&image', 'logs:optimizer.total_norm']
[I 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.76: posix.uname_result(sysname='Linux', nodename='DGX', release='5.4.0-124-generic', version='#140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022', machine='x86_64')
[I 2022-10-28 16:53:27] ...ytorch/bootstrap/run.py.80: Available GPUs: [0]
[I 2022-10-28 16:53:27] ...strap/engines/factory.py.9: Creating engine...
[I 2022-10-28 16:53:27] ...trap/datasets/factory.py.8: Creating dataset...
[I 2022-10-28 16:53:28] ...m/datasets/recipe1m.py.152: Load VCs...
[I 2022-10-28 16:53:29] ...m/datasets/recipe1m.py.165: randkw_p... None
[I 2022-10-28 16:54:32] ...m/datasets/recipe1m.py.310: recipe elements to remove: None
[I 2022-10-28 16:54:32] ...rap/datasets/factory.py.21: Evaluation will take place on testset (51303 items)
[I 2022-10-28 16:54:32] ...strap/models/factory.py.21: Creating model...
[I 2022-10-28 16:54:33] ...e_networks/networks.py.203: Loading CLIP Visual model...
[I 2022-10-28 16:54:36] ...e_networks/networks.py.145: cuda
[I 2022-10-28 16:54:37] ...networks/tfood_vicha.py.49: Load VCs encoder...
[I 2022-10-28 16:54:40] ...els/metrics/trijoint.py.46: self.save_ids False
[I 2022-10-28 16:54:40] ...strap/models/factory.py.44: Enabling CUDA mode...
[I 2022-10-28 16:54:42] ...etworks/tfood_vicha.py.184: training aux kw modules
[I 2022-10-28 16:54:42] ...etworks/tfood_vicha.py.184: training aux kw modules
[I 2022-10-28 16:54:42] ...otstrap/views/factory.py.8: Creating views...
[I 2022-10-28 16:54:42] ...strap/engines/engine.py.98: Loading best_eval_epoch.metric.recall_at_1_im2recipe_mean checkpoint
[I 2022-10-28 16:54:42] ...trap/engines/engine.py.440: Loading model...
[I 2022-10-28 16:54:43] ...trap/engines/engine.py.453: Loading engine...
[I 2022-10-28 16:54:43] ...trap/engines/engine.py.108: Launching evaluation procedures
[I 2022-10-28 16:54:43] ...trap/engines/engine.py.268: Evaluating model on testset for epoch 116
[I 2022-10-28 16:54:43] ...m/datasets/recipe1m.py.400: Dataset will be sampled with "random" batch_sampler.
[I 2022-10-28 16:54:46] ...trap/engines/engine.py.321: eval: epoch 116 | batch 0/512
[I 2022-10-28 16:54:46] ...trap/engines/engine.py.325:       elapsed: 0:00:02 | left: 0:24:19
[I 2022-10-28 16:54:46] ...trap/engines/engine.py.326:       process: 2.84966 | load: 1.61681
[I 2022-10-28 16:54:49] ...trap/engines/engine.py.321: eval: epoch 116 | batch 10/512
[I 2022-10-28 16:54:49] ...trap/engines/engine.py.325:       elapsed: 0:00:06 | left: 0:05:02
[I 2022-10-28 16:54:49] ...trap/engines/engine.py.326:       process: 0.32487 | load: 0.00015
[I 2022-10-28 16:54:53] ...trap/engines/engine.py.321: eval: epoch 116 | batch 20/512
[I 2022-10-28 16:54:53] ...trap/engines/engine.py.325:       elapsed: 0:00:09 | left: 0:02:55
[I 2022-10-28 16:54:53] ...trap/engines/engine.py.326:       process: 0.32449 | load: 0.00016
[I 2022-10-28 16:54:56] ...trap/engines/engine.py.321: eval: epoch 116 | batch 30/512
[I 2022-10-28 16:54:56] ...trap/engines/engine.py.325:       elapsed: 0:00:12 | left: 0:02:39
[I 2022-10-28 16:54:56] ...trap/engines/engine.py.326:       process: 0.32568 | load: 0.00016
[I 2022-10-28 16:54:59] ...trap/engines/engine.py.321: eval: epoch 116 | batch 40/512
[I 2022-10-28 16:54:59] ...trap/engines/engine.py.325:       elapsed: 0:00:16 | left: 0:02:39
[I 2022-10-28 16:54:59] ...trap/engines/engine.py.326:       process: 0.36203 | load: 0.00018
[I 2022-10-28 16:55:03] ...trap/engines/engine.py.321: eval: epoch 116 | batch 50/512
[I 2022-10-28 16:55:03] ...trap/engines/engine.py.325:       elapsed: 0:00:19 | left: 0:02:32
[I 2022-10-28 16:55:03] ...trap/engines/engine.py.326:       process: 0.32917 | load: 0.00020
[I 2022-10-28 16:55:06] ...trap/engines/engine.py.321: eval: epoch 116 | batch 60/512
[I 2022-10-28 16:55:06] ...trap/engines/engine.py.325:       elapsed: 0:00:22 | left: 0:02:30
[I 2022-10-28 16:55:06] ...trap/engines/engine.py.326:       process: 0.32581 | load: 0.00018
[I 2022-10-28 16:55:09] ...trap/engines/engine.py.321: eval: epoch 116 | batch 70/512
[I 2022-10-28 16:55:09] ...trap/engines/engine.py.325:       elapsed: 0:00:26 | left: 0:02:26
[I 2022-10-28 16:55:09] ...trap/engines/engine.py.326:       process: 0.33157 | load: 0.00017
[I 2022-10-28 16:55:13] ...trap/engines/engine.py.321: eval: epoch 116 | batch 80/512
[I 2022-10-28 16:55:13] ...trap/engines/engine.py.325:       elapsed: 0:00:29 | left: 0:02:21
[I 2022-10-28 16:55:13] ...trap/engines/engine.py.326:       process: 0.32448 | load: 0.00016
[I 2022-10-28 16:55:16] ...trap/engines/engine.py.321: eval: epoch 116 | batch 90/512
[I 2022-10-28 16:55:16] ...trap/engines/engine.py.325:       elapsed: 0:00:32 | left: 0:02:18
[I 2022-10-28 16:55:16] ...trap/engines/engine.py.326:       process: 0.32901 | load: 0.00024
[I 2022-10-28 16:55:19] ...trap/engines/engine.py.321: eval: epoch 116 | batch 100/512
[I 2022-10-28 16:55:19] ...trap/engines/engine.py.325:       elapsed: 0:00:36 | left: 0:02:17
[I 2022-10-28 16:55:19] ...trap/engines/engine.py.326:       process: 0.33389 | load: 0.00017
[I 2022-10-28 16:55:23] ...trap/engines/engine.py.321: eval: epoch 116 | batch 110/512
[I 2022-10-28 16:55:23] ...trap/engines/engine.py.325:       elapsed: 0:00:39 | left: 0:02:13
[I 2022-10-28 16:55:23] ...trap/engines/engine.py.326:       process: 0.33111 | load: 0.00016
[I 2022-10-28 16:55:26] ...trap/engines/engine.py.321: eval: epoch 116 | batch 120/512
[I 2022-10-28 16:55:26] ...trap/engines/engine.py.325:       elapsed: 0:00:42 | left: 0:02:10
[I 2022-10-28 16:55:26] ...trap/engines/engine.py.326:       process: 0.33275 | load: 0.00019
[I 2022-10-28 16:55:29] ...trap/engines/engine.py.321: eval: epoch 116 | batch 130/512
[I 2022-10-28 16:55:29] ...trap/engines/engine.py.325:       elapsed: 0:00:46 | left: 0:02:09
[I 2022-10-28 16:55:29] ...trap/engines/engine.py.326:       process: 0.33229 | load: 0.00016
[I 2022-10-28 16:55:33] ...trap/engines/engine.py.321: eval: epoch 116 | batch 140/512
[I 2022-10-28 16:55:33] ...trap/engines/engine.py.325:       elapsed: 0:00:49 | left: 0:02:02
[I 2022-10-28 16:55:33] ...trap/engines/engine.py.326:       process: 0.32871 | load: 0.00016
[I 2022-10-28 16:55:36] ...trap/engines/engine.py.321: eval: epoch 116 | batch 150/512
[I 2022-10-28 16:55:36] ...trap/engines/engine.py.325:       elapsed: 0:00:52 | left: 0:02:00
[I 2022-10-28 16:55:36] ...trap/engines/engine.py.326:       process: 0.33116 | load: 0.00016
[I 2022-10-28 16:55:39] ...trap/engines/engine.py.321: eval: epoch 116 | batch 160/512
[I 2022-10-28 16:55:39] ...trap/engines/engine.py.325:       elapsed: 0:00:56 | left: 0:01:58
[I 2022-10-28 16:55:39] ...trap/engines/engine.py.326:       process: 0.33182 | load: 0.00017
[I 2022-10-28 16:55:43] ...trap/engines/engine.py.321: eval: epoch 116 | batch 170/512
[I 2022-10-28 16:55:43] ...trap/engines/engine.py.325:       elapsed: 0:00:59 | left: 0:01:53
[I 2022-10-28 16:55:43] ...trap/engines/engine.py.326:       process: 0.34996 | load: 0.00016
[I 2022-10-28 16:55:46] ...trap/engines/engine.py.321: eval: epoch 116 | batch 180/512
[I 2022-10-28 16:55:46] ...trap/engines/engine.py.325:       elapsed: 0:01:02 | left: 0:01:50
[I 2022-10-28 16:55:46] ...trap/engines/engine.py.326:       process: 0.32424 | load: 0.00021
[I 2022-10-28 16:55:49] ...trap/engines/engine.py.321: eval: epoch 116 | batch 190/512
[I 2022-10-28 16:55:49] ...trap/engines/engine.py.325:       elapsed: 0:01:06 | left: 0:01:49
[I 2022-10-28 16:55:49] ...trap/engines/engine.py.326:       process: 0.33732 | load: 0.00017
[I 2022-10-28 16:55:53] ...trap/engines/engine.py.321: eval: epoch 116 | batch 200/512
[I 2022-10-28 16:55:53] ...trap/engines/engine.py.325:       elapsed: 0:01:09 | left: 0:01:43
[I 2022-10-28 16:55:53] ...trap/engines/engine.py.326:       process: 0.33407 | load: 0.00017
[I 2022-10-28 16:55:56] ...trap/engines/engine.py.321: eval: epoch 116 | batch 210/512
[I 2022-10-28 16:55:56] ...trap/engines/engine.py.325:       elapsed: 0:01:12 | left: 0:01:40
[I 2022-10-28 16:55:56] ...trap/engines/engine.py.326:       process: 0.32900 | load: 0.00016
[I 2022-10-28 16:55:59] ...trap/engines/engine.py.321: eval: epoch 116 | batch 220/512
[I 2022-10-28 16:55:59] ...trap/engines/engine.py.325:       elapsed: 0:01:16 | left: 0:01:37
[I 2022-10-28 16:55:59] ...trap/engines/engine.py.326:       process: 0.33213 | load: 0.00016
[I 2022-10-28 16:56:03] ...trap/engines/engine.py.321: eval: epoch 116 | batch 230/512
[I 2022-10-28 16:56:03] ...trap/engines/engine.py.325:       elapsed: 0:01:19 | left: 0:01:33
[I 2022-10-28 16:56:03] ...trap/engines/engine.py.326:       process: 0.33157 | load: 0.00015
[I 2022-10-28 16:56:06] ...trap/engines/engine.py.321: eval: epoch 116 | batch 240/512
[I 2022-10-28 16:56:06] ...trap/engines/engine.py.325:       elapsed: 0:01:22 | left: 0:01:30
[I 2022-10-28 16:56:06] ...trap/engines/engine.py.326:       process: 0.33132 | load: 0.00017
[I 2022-10-28 16:56:09] ...trap/engines/engine.py.321: eval: epoch 116 | batch 250/512
[I 2022-10-28 16:56:09] ...trap/engines/engine.py.325:       elapsed: 0:01:26 | left: 0:01:28
[I 2022-10-28 16:56:09] ...trap/engines/engine.py.326:       process: 0.33338 | load: 0.00017
[I 2022-10-28 16:56:13] ...trap/engines/engine.py.321: eval: epoch 116 | batch 260/512
[I 2022-10-28 16:56:13] ...trap/engines/engine.py.325:       elapsed: 0:01:29 | left: 0:01:23
[I 2022-10-28 16:56:13] ...trap/engines/engine.py.326:       process: 0.32965 | load: 0.00017
[I 2022-10-28 16:56:16] ...trap/engines/engine.py.321: eval: epoch 116 | batch 270/512
[I 2022-10-28 16:56:16] ...trap/engines/engine.py.325:       elapsed: 0:01:32 | left: 0:01:20
[I 2022-10-28 16:56:16] ...trap/engines/engine.py.326:       process: 0.33650 | load: 0.00016
[I 2022-10-28 16:56:19] ...trap/engines/engine.py.321: eval: epoch 116 | batch 280/512
[I 2022-10-28 16:56:19] ...trap/engines/engine.py.325:       elapsed: 0:01:36 | left: 0:01:18
[I 2022-10-28 16:56:19] ...trap/engines/engine.py.326:       process: 0.34233 | load: 0.00019
[I 2022-10-28 16:56:23] ...trap/engines/engine.py.321: eval: epoch 116 | batch 290/512
[I 2022-10-28 16:56:23] ...trap/engines/engine.py.325:       elapsed: 0:01:39 | left: 0:01:13
[I 2022-10-28 16:56:23] ...trap/engines/engine.py.326:       process: 0.32720 | load: 0.00018
[I 2022-10-28 16:56:26] ...trap/engines/engine.py.321: eval: epoch 116 | batch 300/512
[I 2022-10-28 16:56:26] ...trap/engines/engine.py.325:       elapsed: 0:01:42 | left: 0:01:10
[I 2022-10-28 16:56:26] ...trap/engines/engine.py.326:       process: 0.33015 | load: 0.00018
[I 2022-10-28 16:56:29] ...trap/engines/engine.py.321: eval: epoch 116 | batch 310/512
[I 2022-10-28 16:56:29] ...trap/engines/engine.py.325:       elapsed: 0:01:46 | left: 0:01:08
[I 2022-10-28 16:56:29] ...trap/engines/engine.py.326:       process: 0.33385 | load: 0.00016
[I 2022-10-28 16:56:33] ...trap/engines/engine.py.321: eval: epoch 116 | batch 320/512
[I 2022-10-28 16:56:33] ...trap/engines/engine.py.325:       elapsed: 0:01:49 | left: 0:01:03
[I 2022-10-28 16:56:33] ...trap/engines/engine.py.326:       process: 0.32999 | load: 0.00017
[I 2022-10-28 16:56:36] ...trap/engines/engine.py.321: eval: epoch 116 | batch 330/512
[I 2022-10-28 16:56:36] ...trap/engines/engine.py.325:       elapsed: 0:01:52 | left: 0:01:00
[I 2022-10-28 16:56:36] ...trap/engines/engine.py.326:       process: 0.32995 | load: 0.00016
[I 2022-10-28 16:56:39] ...trap/engines/engine.py.321: eval: epoch 116 | batch 340/512
[I 2022-10-28 16:56:39] ...trap/engines/engine.py.325:       elapsed: 0:01:56 | left: 0:00:58
[I 2022-10-28 16:56:39] ...trap/engines/engine.py.326:       process: 0.33383 | load: 0.00017
[I 2022-10-28 16:56:43] ...trap/engines/engine.py.321: eval: epoch 116 | batch 350/512
[I 2022-10-28 16:56:43] ...trap/engines/engine.py.325:       elapsed: 0:01:59 | left: 0:00:53
[I 2022-10-28 16:56:43] ...trap/engines/engine.py.326:       process: 0.33086 | load: 0.00016
[I 2022-10-28 16:56:46] ...trap/engines/engine.py.321: eval: epoch 116 | batch 360/512
[I 2022-10-28 16:56:46] ...trap/engines/engine.py.325:       elapsed: 0:02:02 | left: 0:00:50
[I 2022-10-28 16:56:46] ...trap/engines/engine.py.326:       process: 0.33087 | load: 0.00020
[I 2022-10-28 16:56:49] ...trap/engines/engine.py.321: eval: epoch 116 | batch 370/512
[I 2022-10-28 16:56:49] ...trap/engines/engine.py.325:       elapsed: 0:02:06 | left: 0:00:47
[I 2022-10-28 16:56:49] ...trap/engines/engine.py.326:       process: 0.33103 | load: 0.00016
[I 2022-10-28 16:56:53] ...trap/engines/engine.py.321: eval: epoch 116 | batch 380/512
[I 2022-10-28 16:56:53] ...trap/engines/engine.py.325:       elapsed: 0:02:09 | left: 0:00:43
[I 2022-10-28 16:56:53] ...trap/engines/engine.py.326:       process: 0.33336 | load: 0.00018
[I 2022-10-28 16:56:56] ...trap/engines/engine.py.321: eval: epoch 116 | batch 390/512
[I 2022-10-28 16:56:56] ...trap/engines/engine.py.325:       elapsed: 0:02:12 | left: 0:00:40
[I 2022-10-28 16:56:56] ...trap/engines/engine.py.326:       process: 0.32996 | load: 0.00018
[I 2022-10-28 16:56:59] ...trap/engines/engine.py.321: eval: epoch 116 | batch 400/512
[I 2022-10-28 16:56:59] ...trap/engines/engine.py.325:       elapsed: 0:02:16 | left: 0:00:36
[I 2022-10-28 16:56:59] ...trap/engines/engine.py.326:       process: 0.28681 | load: 0.00017
[I 2022-10-28 16:57:02] ...trap/engines/engine.py.321: eval: epoch 116 | batch 410/512
[I 2022-10-28 16:57:02] ...trap/engines/engine.py.325:       elapsed: 0:02:18 | left: 0:00:29
[I 2022-10-28 16:57:02] ...trap/engines/engine.py.326:       process: 0.28631 | load: 0.00023
[I 2022-10-28 16:57:05] ...trap/engines/engine.py.321: eval: epoch 116 | batch 420/512
[I 2022-10-28 16:57:05] ...trap/engines/engine.py.325:       elapsed: 0:02:21 | left: 0:00:26
[I 2022-10-28 16:57:05] ...trap/engines/engine.py.326:       process: 0.28400 | load: 0.00018
[I 2022-10-28 16:57:08] ...trap/engines/engine.py.321: eval: epoch 116 | batch 430/512
[I 2022-10-28 16:57:08] ...trap/engines/engine.py.325:       elapsed: 0:02:24 | left: 0:00:23
[I 2022-10-28 16:57:08] ...trap/engines/engine.py.326:       process: 0.28722 | load: 0.00017
[I 2022-10-28 16:57:11] ...trap/engines/engine.py.321: eval: epoch 116 | batch 440/512
[I 2022-10-28 16:57:11] ...trap/engines/engine.py.325:       elapsed: 0:02:27 | left: 0:00:20
[I 2022-10-28 16:57:11] ...trap/engines/engine.py.326:       process: 0.28722 | load: 0.00017
[I 2022-10-28 16:57:14] ...trap/engines/engine.py.321: eval: epoch 116 | batch 450/512
[I 2022-10-28 16:57:14] ...trap/engines/engine.py.325:       elapsed: 0:02:30 | left: 0:00:17
[I 2022-10-28 16:57:14] ...trap/engines/engine.py.326:       process: 0.28250 | load: 0.00024
[I 2022-10-28 16:57:17] ...trap/engines/engine.py.321: eval: epoch 116 | batch 460/512
[I 2022-10-28 16:57:17] ...trap/engines/engine.py.325:       elapsed: 0:02:33 | left: 0:00:15
[I 2022-10-28 16:57:17] ...trap/engines/engine.py.326:       process: 0.30037 | load: 0.00019
[I 2022-10-28 16:57:20] ...trap/engines/engine.py.321: eval: epoch 116 | batch 470/512
[I 2022-10-28 16:57:20] ...trap/engines/engine.py.325:       elapsed: 0:02:36 | left: 0:00:12
[I 2022-10-28 16:57:20] ...trap/engines/engine.py.326:       process: 0.28239 | load: 0.00019
[I 2022-10-28 16:57:22] ...trap/engines/engine.py.321: eval: epoch 116 | batch 480/512
[I 2022-10-28 16:57:22] ...trap/engines/engine.py.325:       elapsed: 0:02:39 | left: 0:00:09
[I 2022-10-28 16:57:22] ...trap/engines/engine.py.326:       process: 0.28706 | load: 0.00018
[I 2022-10-28 16:57:25] ...trap/engines/engine.py.321: eval: epoch 116 | batch 490/512
[I 2022-10-28 16:57:25] ...trap/engines/engine.py.325:       elapsed: 0:02:42 | left: 0:00:06
[I 2022-10-28 16:57:25] ...trap/engines/engine.py.326:       process: 0.29343 | load: 0.00200
[I 2022-10-28 16:57:28] ...trap/engines/engine.py.321: eval: epoch 116 | batch 500/512
[I 2022-10-28 16:57:28] ...trap/engines/engine.py.325:       elapsed: 0:02:44 | left: 0:00:03
[I 2022-10-28 16:57:28] ...trap/engines/engine.py.326:       process: 0.28613 | load: 0.00018
[I 2022-10-28 16:57:31] ...trap/engines/engine.py.321: eval: epoch 116 | batch 510/512
[I 2022-10-28 16:57:31] ...trap/engines/engine.py.325:       elapsed: 0:02:47 | left: 0:00:00
[I 2022-10-28 16:57:31] ...trap/engines/engine.py.326:       process: 0.28129 | load: 0.00010
[S 2022-10-28 16:57:32] ...trap/engines/engine.py.113: eval_epoch.epoch: 116
[I 2022-10-28 16:57:32] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 1 x 40000 matchs
[S 2022-10-28 17:05:40] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 4.00
[S 2022-10-28 17:05:40] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 4.00
[S 2022-10-28 17:05:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.301700
[S 2022-10-28 17:05:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.549650
[S 2022-10-28 17:05:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.654025
[S 2022-10-28 17:05:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.287875
[S 2022-10-28 17:05:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.537075
[S 2022-10-28 17:05:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.641475
[S 2022-10-28 17:05:40] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000
[S 2022-10-28 17:05:40] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000
[S 2022-10-28 17:05:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.000000
[S 2022-10-28 17:05:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.000000
[S 2022-10-28 17:05:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.000000
[S 2022-10-28 17:05:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.000000
[S 2022-10-28 17:05:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.000000
[S 2022-10-28 17:05:40] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.000000
[I 2022-10-28 17:05:42] ...trap/engines/engine.py.115: Ending evaluation procedures
[W 2022-10-28 17:05:43] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json'
[W 2022-10-28 17:05:43] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[W 2022-10-28 17:05:43] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[I 2022-10-28 17:05:47] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html
[I 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.73: Saving environment info
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75: options: 
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:   dataset:
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     aux_kw_path:
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       test: /home/mshukor/data/recipe1m/clip_da/layer1_test_titles_kw.json
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       train: /home/mshukor/data/recipe1m/clip_da/layer1_train_titles_kw.json
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       val: /home/mshukor/data/recipe1m/clip_da/layer1_val_titles_kw.json
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     batch_sampler: triplet_classif
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     batch_size: 100
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     database: lmdb
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     dir: /home/mshukor/data/recipe1m
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     eval_split: test
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     freq_mismatch: 0.0
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     image_from: database
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     import: recipe1m.datasets.factory
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     interchange_ingrd_instr: None
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     kw_path:
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       test: /home/mshukor/data/recipe1m/clip_da/layer1_test_ingr_kw.json
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       train: /home/mshukor/data/recipe1m/clip_da/layer1_train_ingr_kw.json
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       val: /home/mshukor/data/recipe1m/clip_da/layer1_val_ingr_kw.json
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     name: recipe1m
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     nb_classes: 1048
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     nb_threads: 4
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     randkw_p: 0.3
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     randkw_p_aux: 0.5
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     remove_list: None
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     tokenized_raw_text: True
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     train_split: None
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     use_vcs: True
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:   engine:
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     name: logger
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     nb_epochs: 120
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     print_freq: 10
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     saving_criteria: ['eval_epoch.metric.med_im2recipe_mean:min', 'eval_epoch.metric.recall_at_1_im2recipe_mean:max']
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:   exp:
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     checkpoint: None
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     dir: /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     resume: best_eval_epoch.metric.recall_at_1_im2recipe_mean
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:   misc:
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     cuda: True
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     data_parrallel: True
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     logs_name: None
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     overrite: False
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     seed: 1338
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:   model:
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     criterion:
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:         tri_cross_encoder: False
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       itm_loss_weight: 1
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       keep_background: False
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       retrieval_strategy:
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:         aggregation: valid
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:         get_tokens: True
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:         margin: 0.05
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:         margin_params:
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:           increment: 0.005
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:           increment_margin: True
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:           max_margin: 0.3
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:         name: triplet
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:         nb_samples: 9999
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:         sampling: max_negative
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:         substrategy: ['IRR', 'RII', 'SIRR', 'SRII']
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:         substrategy_weights: [1.0, 1.0, 0.1, 0.1]
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     import: recipe1m.models.factory
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     metric:
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       k_test: 10
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       nb_bags: 1
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       nb_matchs_per_bag: 50000
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       retrieval_dir: /tmp/recipe1m
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       save_ids: False
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       trijoint: True
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     name: trijoint
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     network:
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       activations: ['tanh', 'normalize']
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       aux_kwords: True
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       aux_kwords_encoder: True
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       avg_concat: True
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       bert_config: /home/mshukor/tfood/recipe1m/models/networks/recipe_networks/config_bert_albef.json
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       cat_pos: 0
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       checkpoint: None
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       class_attention: False
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       class_attention_fus: False
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       cls_token: False
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       cls_token_fus: False
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       cross_decoder_image: True
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       cross_decoder_recipe: True
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:         class_attention: False
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:         cls_token: False
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:         cross_attention: False
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:         cross_decoder: True
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:         double_decoder: False
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:         get_tokens: False
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:         load_backbones: False
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:         n_heads: 4
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:         n_layers: 4
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:         query_img: False
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       cross_transformer: False
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       dim_emb: 1024
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       dim_image_out: 2048
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       dim_ingr_out: 300
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       dim_instr_in: 1024
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       dim_instr_out: 1024
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       freeze_im: False
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       freeze_rec: False
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       get_tokens: True
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       get_tokens_cross_decoder_recipe: True
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       hidden_size: 512
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       image_backbone_name: clip
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       image_res: 224
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       kwords_same_level: False
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       last_cat_aux: True
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       n_heads: 4
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       n_heads_cross: 4
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       n_heads_cross_image: 4
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       n_heads_single: 4
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       n_layers: 2
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       n_layers_cross: 2
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       n_layers_cross_image: 1
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       n_layers_single: 2
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       num_hidden_layers_kw: 2
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       old: True
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       path_ingrs: /home/mshukor/data/recipe1m/text/vocab.pkl
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       path_vocab: /home/mshukor/data/recipe1m/text/vocab_all.txt
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       recipe_encoder: tfoodvicha-transformer
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       remove_additional_embedding: ['proj_image']
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       text_encoder: bert-base-uncased
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       vision_width: 768
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       vit_all_tokens: True
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       with_ingrs: True
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       with_instrs: True
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:       with_titles: True
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     with_classif: False
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:   optimizer:
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     batch_size_factor: None
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     clip_grad: 8.0
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     import: recipe1m.optimizers.factory
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     lr: 1e-05
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     lr_img: 1e-06
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     name: trijoint_fixed_fine_tune
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:     switch_epoch: 20
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:   path_opts: /home/mshukor/tfood/recipe1m/options/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_noimgproj.yaml
[S 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.75:   view: ['logs:train_epoch.loss', 'logs:train_epoch.bad_pairs', 'logs:eval_epoch.metric.med_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_1_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_5_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_10_im2recipe_mean', 'logs:eval_epoch.metric.med_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_1_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_5_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_10_recipe2im_mean', 'logs:optimizer.is_optimizer_recipe&image', 'logs:optimizer.total_norm']
[I 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.76: posix.uname_result(sysname='Linux', nodename='DGX', release='5.4.0-124-generic', version='#140-Ubuntu SMP Thu Aug 4 02:23:37 UTC 2022', machine='x86_64')
[I 2022-10-28 17:06:32] ...ytorch/bootstrap/run.py.80: Available GPUs: [0]
[I 2022-10-28 17:06:32] ...strap/engines/factory.py.9: Creating engine...
[I 2022-10-28 17:06:32] ...trap/datasets/factory.py.8: Creating dataset...
[I 2022-10-28 17:06:34] ...m/datasets/recipe1m.py.152: Load VCs...
[I 2022-10-28 17:06:35] ...m/datasets/recipe1m.py.165: randkw_p... None
[I 2022-10-28 17:07:38] ...m/datasets/recipe1m.py.310: recipe elements to remove: None
[I 2022-10-28 17:07:38] ...rap/datasets/factory.py.21: Evaluation will take place on testset (51303 items)
[I 2022-10-28 17:07:38] ...strap/models/factory.py.21: Creating model...
[I 2022-10-28 17:07:39] ...e_networks/networks.py.203: Loading CLIP Visual model...
[I 2022-10-28 17:07:42] ...e_networks/networks.py.145: cuda
[I 2022-10-28 17:07:43] ...networks/tfood_vicha.py.49: Load VCs encoder...
[I 2022-10-28 17:07:46] ...els/metrics/trijoint.py.46: self.save_ids False
[I 2022-10-28 17:07:46] ...strap/models/factory.py.44: Enabling CUDA mode...
[I 2022-10-28 17:07:48] ...etworks/tfood_vicha.py.184: training aux kw modules
[I 2022-10-28 17:07:48] ...etworks/tfood_vicha.py.184: training aux kw modules
[I 2022-10-28 17:07:48] ...otstrap/views/factory.py.8: Creating views...
[I 2022-10-28 17:07:49] ...strap/engines/engine.py.98: Loading best_eval_epoch.metric.recall_at_1_im2recipe_mean checkpoint
[I 2022-10-28 17:07:49] ...trap/engines/engine.py.440: Loading model...
[I 2022-10-28 17:07:50] ...trap/engines/engine.py.453: Loading engine...
[I 2022-10-28 17:07:50] ...trap/engines/engine.py.108: Launching evaluation procedures
[I 2022-10-28 17:07:50] ...trap/engines/engine.py.268: Evaluating model on testset for epoch 116
[I 2022-10-28 17:07:50] ...m/datasets/recipe1m.py.400: Dataset will be sampled with "random" batch_sampler.
[I 2022-10-28 17:07:52] ...trap/engines/engine.py.321: eval: epoch 116 | batch 0/512
[I 2022-10-28 17:07:52] ...trap/engines/engine.py.325:       elapsed: 0:00:02 | left: 0:23:32
[I 2022-10-28 17:07:52] ...trap/engines/engine.py.326:       process: 2.75917 | load: 1.50015
[I 2022-10-28 17:07:56] ...trap/engines/engine.py.321: eval: epoch 116 | batch 10/512
[I 2022-10-28 17:07:56] ...trap/engines/engine.py.325:       elapsed: 0:00:06 | left: 0:04:56
[I 2022-10-28 17:07:56] ...trap/engines/engine.py.326:       process: 0.32549 | load: 0.00015
[I 2022-10-28 17:07:59] ...trap/engines/engine.py.321: eval: epoch 116 | batch 20/512
[I 2022-10-28 17:07:59] ...trap/engines/engine.py.325:       elapsed: 0:00:09 | left: 0:02:56
[I 2022-10-28 17:07:59] ...trap/engines/engine.py.326:       process: 0.32127 | load: 0.00015
[I 2022-10-28 17:08:02] ...trap/engines/engine.py.321: eval: epoch 116 | batch 30/512
[I 2022-10-28 17:08:02] ...trap/engines/engine.py.325:       elapsed: 0:00:12 | left: 0:02:43
[I 2022-10-28 17:08:02] ...trap/engines/engine.py.326:       process: 0.32220 | load: 0.00024
[I 2022-10-28 17:08:06] ...trap/engines/engine.py.321: eval: epoch 116 | batch 40/512
[I 2022-10-28 17:08:06] ...trap/engines/engine.py.325:       elapsed: 0:00:16 | left: 0:02:38
[I 2022-10-28 17:08:06] ...trap/engines/engine.py.326:       process: 0.33045 | load: 0.00016
[I 2022-10-28 17:08:09] ...trap/engines/engine.py.321: eval: epoch 116 | batch 50/512
[I 2022-10-28 17:08:09] ...trap/engines/engine.py.325:       elapsed: 0:00:19 | left: 0:02:40
[I 2022-10-28 17:08:09] ...trap/engines/engine.py.326:       process: 0.32889 | load: 0.00021
[I 2022-10-28 17:08:12] ...trap/engines/engine.py.321: eval: epoch 116 | batch 60/512
[I 2022-10-28 17:08:12] ...trap/engines/engine.py.325:       elapsed: 0:00:22 | left: 0:02:30
[I 2022-10-28 17:08:12] ...trap/engines/engine.py.326:       process: 0.33436 | load: 0.00016
[I 2022-10-28 17:08:16] ...trap/engines/engine.py.321: eval: epoch 116 | batch 70/512
[I 2022-10-28 17:08:16] ...trap/engines/engine.py.325:       elapsed: 0:00:26 | left: 0:02:26
[I 2022-10-28 17:08:16] ...trap/engines/engine.py.326:       process: 0.33192 | load: 0.00014
[I 2022-10-28 17:08:19] ...trap/engines/engine.py.321: eval: epoch 116 | batch 80/512
[I 2022-10-28 17:08:19] ...trap/engines/engine.py.325:       elapsed: 0:00:29 | left: 0:02:23
[I 2022-10-28 17:08:19] ...trap/engines/engine.py.326:       process: 0.32432 | load: 0.00015
[I 2022-10-28 17:08:22] ...trap/engines/engine.py.321: eval: epoch 116 | batch 90/512
[I 2022-10-28 17:08:22] ...trap/engines/engine.py.325:       elapsed: 0:00:32 | left: 0:02:19
[I 2022-10-28 17:08:22] ...trap/engines/engine.py.326:       process: 0.33765 | load: 0.00023
[I 2022-10-28 17:08:26] ...trap/engines/engine.py.321: eval: epoch 116 | batch 100/512
[I 2022-10-28 17:08:26] ...trap/engines/engine.py.325:       elapsed: 0:00:36 | left: 0:02:17
[I 2022-10-28 17:08:26] ...trap/engines/engine.py.326:       process: 0.34056 | load: 0.00018
[I 2022-10-28 17:08:29] ...trap/engines/engine.py.321: eval: epoch 116 | batch 110/512
[I 2022-10-28 17:08:29] ...trap/engines/engine.py.325:       elapsed: 0:00:39 | left: 0:02:18
[I 2022-10-28 17:08:29] ...trap/engines/engine.py.326:       process: 0.32797 | load: 0.00016
[I 2022-10-28 17:08:32] ...trap/engines/engine.py.321: eval: epoch 116 | batch 120/512
[I 2022-10-28 17:08:32] ...trap/engines/engine.py.325:       elapsed: 0:00:42 | left: 0:02:12
[I 2022-10-28 17:08:32] ...trap/engines/engine.py.326:       process: 0.35127 | load: 0.00016
[I 2022-10-28 17:08:36] ...trap/engines/engine.py.321: eval: epoch 116 | batch 130/512
[I 2022-10-28 17:08:36] ...trap/engines/engine.py.325:       elapsed: 0:00:46 | left: 0:02:06
[I 2022-10-28 17:08:36] ...trap/engines/engine.py.326:       process: 0.32618 | load: 0.00015
[I 2022-10-28 17:08:39] ...trap/engines/engine.py.321: eval: epoch 116 | batch 140/512
[I 2022-10-28 17:08:39] ...trap/engines/engine.py.325:       elapsed: 0:00:49 | left: 0:02:04
[I 2022-10-28 17:08:39] ...trap/engines/engine.py.326:       process: 0.32889 | load: 0.00016
[I 2022-10-28 17:08:42] ...trap/engines/engine.py.321: eval: epoch 116 | batch 150/512
[I 2022-10-28 17:08:42] ...trap/engines/engine.py.325:       elapsed: 0:00:52 | left: 0:02:00
[I 2022-10-28 17:08:42] ...trap/engines/engine.py.326:       process: 0.33098 | load: 0.00016
[I 2022-10-28 17:08:46] ...trap/engines/engine.py.321: eval: epoch 116 | batch 160/512
[I 2022-10-28 17:08:46] ...trap/engines/engine.py.325:       elapsed: 0:00:56 | left: 0:02:00
[I 2022-10-28 17:08:46] ...trap/engines/engine.py.326:       process: 0.33144 | load: 0.00016
[I 2022-10-28 17:08:49] ...trap/engines/engine.py.321: eval: epoch 116 | batch 170/512
[I 2022-10-28 17:08:49] ...trap/engines/engine.py.325:       elapsed: 0:00:59 | left: 0:01:53
[I 2022-10-28 17:08:49] ...trap/engines/engine.py.326:       process: 0.33145 | load: 0.00016
[I 2022-10-28 17:08:53] ...trap/engines/engine.py.321: eval: epoch 116 | batch 180/512
[I 2022-10-28 17:08:53] ...trap/engines/engine.py.325:       elapsed: 0:01:03 | left: 0:01:50
[I 2022-10-28 17:08:53] ...trap/engines/engine.py.326:       process: 0.32569 | load: 0.00016
[I 2022-10-28 17:08:56] ...trap/engines/engine.py.321: eval: epoch 116 | batch 190/512
[I 2022-10-28 17:08:56] ...trap/engines/engine.py.325:       elapsed: 0:01:06 | left: 0:01:50
[I 2022-10-28 17:08:56] ...trap/engines/engine.py.326:       process: 0.34558 | load: 0.00017
[I 2022-10-28 17:08:59] ...trap/engines/engine.py.321: eval: epoch 116 | batch 200/512
[I 2022-10-28 17:08:59] ...trap/engines/engine.py.325:       elapsed: 0:01:09 | left: 0:01:43
[I 2022-10-28 17:08:59] ...trap/engines/engine.py.326:       process: 0.33374 | load: 0.00016
[I 2022-10-28 17:09:03] ...trap/engines/engine.py.321: eval: epoch 116 | batch 210/512
[I 2022-10-28 17:09:03] ...trap/engines/engine.py.325:       elapsed: 0:01:13 | left: 0:01:39
[I 2022-10-28 17:09:03] ...trap/engines/engine.py.326:       process: 0.32890 | load: 0.00015
[I 2022-10-28 17:09:06] ...trap/engines/engine.py.321: eval: epoch 116 | batch 220/512
[I 2022-10-28 17:09:06] ...trap/engines/engine.py.325:       elapsed: 0:01:16 | left: 0:01:37
[I 2022-10-28 17:09:06] ...trap/engines/engine.py.326:       process: 0.33168 | load: 0.00017
[I 2022-10-28 17:09:09] ...trap/engines/engine.py.321: eval: epoch 116 | batch 230/512
[I 2022-10-28 17:09:09] ...trap/engines/engine.py.325:       elapsed: 0:01:19 | left: 0:01:34
[I 2022-10-28 17:09:09] ...trap/engines/engine.py.326:       process: 0.34575 | load: 0.00016
[I 2022-10-28 17:09:13] ...trap/engines/engine.py.321: eval: epoch 116 | batch 240/512
[I 2022-10-28 17:09:13] ...trap/engines/engine.py.325:       elapsed: 0:01:23 | left: 0:01:30
[I 2022-10-28 17:09:13] ...trap/engines/engine.py.326:       process: 0.33218 | load: 0.00016
[I 2022-10-28 17:09:16] ...trap/engines/engine.py.321: eval: epoch 116 | batch 250/512
[I 2022-10-28 17:09:16] ...trap/engines/engine.py.325:       elapsed: 0:01:26 | left: 0:01:27
[I 2022-10-28 17:09:16] ...trap/engines/engine.py.326:       process: 0.33652 | load: 0.00023
[I 2022-10-28 17:09:19] ...trap/engines/engine.py.321: eval: epoch 116 | batch 260/512
[I 2022-10-28 17:09:19] ...trap/engines/engine.py.325:       elapsed: 0:01:29 | left: 0:01:23
[I 2022-10-28 17:09:19] ...trap/engines/engine.py.326:       process: 0.32920 | load: 0.00017
[I 2022-10-28 17:09:23] ...trap/engines/engine.py.321: eval: epoch 116 | batch 270/512
[I 2022-10-28 17:09:23] ...trap/engines/engine.py.325:       elapsed: 0:01:33 | left: 0:01:20
[I 2022-10-28 17:09:23] ...trap/engines/engine.py.326:       process: 0.34076 | load: 0.00016
[I 2022-10-28 17:09:26] ...trap/engines/engine.py.321: eval: epoch 116 | batch 280/512
[I 2022-10-28 17:09:26] ...trap/engines/engine.py.325:       elapsed: 0:01:36 | left: 0:01:18
[I 2022-10-28 17:09:26] ...trap/engines/engine.py.326:       process: 0.33174 | load: 0.00016
[I 2022-10-28 17:09:29] ...trap/engines/engine.py.321: eval: epoch 116 | batch 290/512
[I 2022-10-28 17:09:29] ...trap/engines/engine.py.325:       elapsed: 0:01:39 | left: 0:01:13
[I 2022-10-28 17:09:29] ...trap/engines/engine.py.326:       process: 0.32643 | load: 0.00017
[I 2022-10-28 17:09:33] ...trap/engines/engine.py.321: eval: epoch 116 | batch 300/512
[I 2022-10-28 17:09:33] ...trap/engines/engine.py.325:       elapsed: 0:01:43 | left: 0:01:10
[I 2022-10-28 17:09:33] ...trap/engines/engine.py.326:       process: 0.32959 | load: 0.00017
[I 2022-10-28 17:09:36] ...trap/engines/engine.py.321: eval: epoch 116 | batch 310/512
[I 2022-10-28 17:09:36] ...trap/engines/engine.py.325:       elapsed: 0:01:46 | left: 0:01:07
[I 2022-10-28 17:09:36] ...trap/engines/engine.py.326:       process: 0.33394 | load: 0.00016
[I 2022-10-28 17:09:39] ...trap/engines/engine.py.321: eval: epoch 116 | batch 320/512
[I 2022-10-28 17:09:39] ...trap/engines/engine.py.325:       elapsed: 0:01:49 | left: 0:01:03
[I 2022-10-28 17:09:39] ...trap/engines/engine.py.326:       process: 0.33436 | load: 0.00016
[I 2022-10-28 17:09:43] ...trap/engines/engine.py.321: eval: epoch 116 | batch 330/512
[I 2022-10-28 17:09:43] ...trap/engines/engine.py.325:       elapsed: 0:01:53 | left: 0:01:00
[I 2022-10-28 17:09:43] ...trap/engines/engine.py.326:       process: 0.33159 | load: 0.00017
[I 2022-10-28 17:09:46] ...trap/engines/engine.py.321: eval: epoch 116 | batch 340/512
[I 2022-10-28 17:09:46] ...trap/engines/engine.py.325:       elapsed: 0:01:56 | left: 0:00:57
[I 2022-10-28 17:09:46] ...trap/engines/engine.py.326:       process: 0.33411 | load: 0.00015
[I 2022-10-28 17:09:49] ...trap/engines/engine.py.321: eval: epoch 116 | batch 350/512
[I 2022-10-28 17:09:49] ...trap/engines/engine.py.325:       elapsed: 0:01:59 | left: 0:00:55
[I 2022-10-28 17:09:49] ...trap/engines/engine.py.326:       process: 0.35185 | load: 0.00019
[I 2022-10-28 17:09:53] ...trap/engines/engine.py.321: eval: epoch 116 | batch 360/512
[I 2022-10-28 17:09:53] ...trap/engines/engine.py.325:       elapsed: 0:02:03 | left: 0:00:50
[I 2022-10-28 17:09:53] ...trap/engines/engine.py.326:       process: 0.33638 | load: 0.00017
[I 2022-10-28 17:09:56] ...trap/engines/engine.py.321: eval: epoch 116 | batch 370/512
[I 2022-10-28 17:09:56] ...trap/engines/engine.py.325:       elapsed: 0:02:06 | left: 0:00:47
[I 2022-10-28 17:09:56] ...trap/engines/engine.py.326:       process: 0.33149 | load: 0.00015
[I 2022-10-28 17:09:59] ...trap/engines/engine.py.321: eval: epoch 116 | batch 380/512
[I 2022-10-28 17:09:59] ...trap/engines/engine.py.325:       elapsed: 0:02:09 | left: 0:00:44
[I 2022-10-28 17:09:59] ...trap/engines/engine.py.326:       process: 0.37002 | load: 0.00016
[I 2022-10-28 17:10:03] ...trap/engines/engine.py.321: eval: epoch 116 | batch 390/512
[I 2022-10-28 17:10:03] ...trap/engines/engine.py.325:       elapsed: 0:02:13 | left: 0:00:40
[I 2022-10-28 17:10:03] ...trap/engines/engine.py.326:       process: 0.33003 | load: 0.00016
[I 2022-10-28 17:10:06] ...trap/engines/engine.py.321: eval: epoch 116 | batch 400/512
[I 2022-10-28 17:10:06] ...trap/engines/engine.py.325:       elapsed: 0:02:16 | left: 0:00:37
[I 2022-10-28 17:10:06] ...trap/engines/engine.py.326:       process: 0.32944 | load: 0.00017
[I 2022-10-28 17:10:09] ...trap/engines/engine.py.321: eval: epoch 116 | batch 410/512
[I 2022-10-28 17:10:09] ...trap/engines/engine.py.325:       elapsed: 0:02:19 | left: 0:00:34
[I 2022-10-28 17:10:09] ...trap/engines/engine.py.326:       process: 0.36998 | load: 0.00016
[I 2022-10-28 17:10:13] ...trap/engines/engine.py.321: eval: epoch 116 | batch 420/512
[I 2022-10-28 17:10:13] ...trap/engines/engine.py.325:       elapsed: 0:02:23 | left: 0:00:30
[I 2022-10-28 17:10:13] ...trap/engines/engine.py.326:       process: 0.33238 | load: 0.00018
[I 2022-10-28 17:10:16] ...trap/engines/engine.py.321: eval: epoch 116 | batch 430/512
[I 2022-10-28 17:10:16] ...trap/engines/engine.py.325:       elapsed: 0:02:26 | left: 0:00:27
[I 2022-10-28 17:10:16] ...trap/engines/engine.py.326:       process: 0.33005 | load: 0.00017
[I 2022-10-28 17:10:19] ...trap/engines/engine.py.321: eval: epoch 116 | batch 440/512
[I 2022-10-28 17:10:19] ...trap/engines/engine.py.325:       elapsed: 0:02:29 | left: 0:00:24
[I 2022-10-28 17:10:19] ...trap/engines/engine.py.326:       process: 0.36221 | load: 0.00016
[I 2022-10-28 17:10:23] ...trap/engines/engine.py.321: eval: epoch 116 | batch 450/512
[I 2022-10-28 17:10:23] ...trap/engines/engine.py.325:       elapsed: 0:02:33 | left: 0:00:20
[I 2022-10-28 17:10:23] ...trap/engines/engine.py.326:       process: 0.32486 | load: 0.00017
[I 2022-10-28 17:10:26] ...trap/engines/engine.py.321: eval: epoch 116 | batch 460/512
[I 2022-10-28 17:10:26] ...trap/engines/engine.py.325:       elapsed: 0:02:36 | left: 0:00:17
[I 2022-10-28 17:10:26] ...trap/engines/engine.py.326:       process: 0.33184 | load: 0.00019
[I 2022-10-28 17:10:29] ...trap/engines/engine.py.321: eval: epoch 116 | batch 470/512
[I 2022-10-28 17:10:29] ...trap/engines/engine.py.325:       elapsed: 0:02:39 | left: 0:00:14
[I 2022-10-28 17:10:29] ...trap/engines/engine.py.326:       process: 0.35747 | load: 0.00015
[I 2022-10-28 17:10:33] ...trap/engines/engine.py.321: eval: epoch 116 | batch 480/512
[I 2022-10-28 17:10:33] ...trap/engines/engine.py.325:       elapsed: 0:02:43 | left: 0:00:10
[I 2022-10-28 17:10:33] ...trap/engines/engine.py.326:       process: 0.33340 | load: 0.00015
[I 2022-10-28 17:10:36] ...trap/engines/engine.py.321: eval: epoch 116 | batch 490/512
[I 2022-10-28 17:10:36] ...trap/engines/engine.py.325:       elapsed: 0:02:46 | left: 0:00:07
[I 2022-10-28 17:10:36] ...trap/engines/engine.py.326:       process: 0.32077 | load: 0.00016
[I 2022-10-28 17:10:39] ...trap/engines/engine.py.321: eval: epoch 116 | batch 500/512
[I 2022-10-28 17:10:39] ...trap/engines/engine.py.325:       elapsed: 0:02:49 | left: 0:00:03
[I 2022-10-28 17:10:39] ...trap/engines/engine.py.326:       process: 0.28244 | load: 0.00015
[I 2022-10-28 17:10:42] ...trap/engines/engine.py.321: eval: epoch 116 | batch 510/512
[I 2022-10-28 17:10:42] ...trap/engines/engine.py.325:       elapsed: 0:02:52 | left: 0:00:00
[I 2022-10-28 17:10:42] ...trap/engines/engine.py.326:       process: 0.28083 | load: 0.00010
[S 2022-10-28 17:10:43] ...trap/engines/engine.py.113: eval_epoch.epoch: 116
[I 2022-10-28 17:10:43] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 1 x 50000 matchs
[S 2022-10-28 17:21:46] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 5.00
[S 2022-10-28 17:21:46] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 5.00
[S 2022-10-28 17:21:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.278880
[S 2022-10-28 17:21:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.518320
[S 2022-10-28 17:21:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.621660
[S 2022-10-28 17:21:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.264760
[S 2022-10-28 17:21:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.506740
[S 2022-10-28 17:21:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.609740
[S 2022-10-28 17:21:46] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000
[S 2022-10-28 17:21:46] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000
[S 2022-10-28 17:21:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.000000
[S 2022-10-28 17:21:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.000000
[S 2022-10-28 17:21:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.000000
[S 2022-10-28 17:21:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.000000
[S 2022-10-28 17:21:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.000000
[S 2022-10-28 17:21:46] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.000000
[I 2022-10-28 17:21:48] ...trap/engines/engine.py.115: Ending evaluation procedures
[W 2022-10-28 17:21:49] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json'
[W 2022-10-28 17:21:49] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[W 2022-10-28 17:21:49] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[I 2022-10-28 17:21:53] ...tstrap/views/plotly.py.125: Plotly view generated in /home/mshukor/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_noclip_nobert/view.html