[I 2022-10-11 20:03:13] ...ytorch/bootstrap/run.py.73: Saving environment info
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75: options: 
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:   dataset:
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     aux_kw_path:
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       test: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_test_titles_kw.json
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       train: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_train_titles_kw.json
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       val: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_val_titles_kw.json
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     batch_sampler: triplet_classif
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     batch_size: 100
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     database: lmdb
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     dir: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     eval_split: test
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     freq_mismatch: 0.0
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     image_from: database
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     import: recipe1m.datasets.factory
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     kw_path:
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       test: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_test_ingr_kw.json
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       train: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_train_ingr_kw.json
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       val: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_val_ingr_kw.json
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     name: recipe1m
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     nb_classes: 1048
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     nb_threads: 4
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     randkw_p: 0.3
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     randkw_p_aux: 0.5
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     tokenized_raw_text: True
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     train_split: None
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     use_vcs: True
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:   engine:
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     name: logger
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     nb_epochs: 120
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     print_freq: 10
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     saving_criteria: ['eval_epoch.metric.med_im2recipe_mean:min', 'eval_epoch.metric.recall_at_1_im2recipe_mean:max']
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:   exp:
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     checkpoint: /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_recipe1m_13m_vlp/ckpt_last_model.pth.tar
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     dir: /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     resume: last
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:   misc:
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     cuda: True
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     data_parrallel: True
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     logs_name: None
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     overrite: False
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     seed: 1338
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:   model:
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     criterion:
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:         tri_cross_encoder: False
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       itm_loss_weight: 1
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       keep_background: False
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       retrieval_strategy:
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:         aggregation: valid
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:         get_tokens: True
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:         margin: 0.05
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:         margin_params:
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:           increment: 0.005
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:           increment_margin: True
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:           max_margin: 0.3
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:         name: triplet
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:         nb_samples: 9999
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:         sampling: max_negative
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:         substrategy: ['IRR', 'RII', 'SIRR', 'SRII']
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:         substrategy_weights: [1.0, 1.0, 0.1, 0.1]
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     import: recipe1m.models.factory
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     metric:
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       k_test: 10
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       nb_bags: 5
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       nb_matchs_per_bag: 10000
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       retrieval_dir: /gpfsscratch/rech/dyf/ugz83ue/tmp/recipe1m
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       trijoint: True
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     name: trijoint
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     network:
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       activations: ['tanh', 'normalize']
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       aux_kwords: True
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       aux_kwords_encoder: True
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       avg_concat: True
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       bert_config: /linkhome/rech/genisi01/ugz83ue/tfood/recipe1m/models/networks/recipe_networks/config_bert_albef.json
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       cat_pos: 0
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       checkpoint: None
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       class_attention: False
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       class_attention_fus: False
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       cls_token: False
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       cls_token_fus: False
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       cross_decoder_image: True
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       cross_decoder_recipe: True
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:         class_attention: False
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:         cls_token: False
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:         cross_attention: False
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:         cross_decoder: True
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:         double_decoder: False
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:         get_tokens: False
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:         load_backbones: False
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:         n_heads: 4
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:         n_layers: 4
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:         query_img: False
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       cross_transformer: False
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       dim_emb: 1024
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       dim_image_out: 2048
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       dim_ingr_out: 300
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       dim_instr_in: 1024
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       dim_instr_out: 1024
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       embedding_names: ['proj_image']
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       freeze_im: False
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       freeze_rec: False
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       get_tokens: True
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       get_tokens_cross_decoder_recipe: True
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       hidden_size: 512
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       image_backbone_name: clip
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       image_res: 224
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       kwords_same_level: False
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       last_cat_aux: True
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       n_heads: 4
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       n_heads_cross: 4
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       n_heads_cross_image: 4
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       n_heads_single: 4
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       n_layers: 2
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       n_layers_cross: 2
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       n_layers_cross_image: 1
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       n_layers_single: 2
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       num_hidden_layers_kw: 2
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       old: True
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       path_ingrs: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/text/vocab.pkl
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       path_vocab: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/text/vocab_all.txt
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       recipe_encoder: tfoodvicha-transformer
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       text_encoder: bert-base-uncased
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       vision_width: 768
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       vit_all_tokens: True
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       with_ingrs: True
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       with_instrs: True
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:       with_titles: True
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     with_classif: False
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:   optimizer:
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     batch_size_factor: None
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     clip_grad: 8.0
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     import: recipe1m.optimizers.factory
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     lr: 1e-05
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     lr_img: 1e-06
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     name: trijoint_fixed_fine_tune
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:     switch_epoch: 20
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:   path_opts: /linkhome/rech/genisi01/ugz83ue/tfood/recipe1m/options/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_noimgproj_keepall.yaml
[S 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.75:   view: ['logs:train_epoch.loss', 'logs:train_epoch.bad_pairs', 'logs:eval_epoch.metric.med_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_1_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_5_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_10_im2recipe_mean', 'logs:eval_epoch.metric.med_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_1_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_5_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_10_recipe2im_mean', 'logs:optimizer.is_optimizer_recipe&image', 'logs:optimizer.total_norm']
[I 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.76: posix.uname_result(sysname='Linux', nodename='jean-zay-ia825', release='4.18.0-305.57.1.el8_4.x86_64', version='#1 SMP Tue Jul 5 17:55:55 EDT 2022', machine='x86_64')
[I 2022-10-11 20:03:14] ...ytorch/bootstrap/run.py.80: Available GPUs: [0]
[I 2022-10-11 20:03:14] ...strap/engines/factory.py.9: Creating engine...
[I 2022-10-11 20:03:14] ...trap/datasets/factory.py.8: Creating dataset...
[I 2022-10-11 20:03:18] ...m/datasets/recipe1m.py.152: Load VCs...
[I 2022-10-11 20:03:18] ...m/datasets/recipe1m.py.165: randkw_p... None
[I 2022-10-11 20:04:20] ...rap/datasets/factory.py.21: Evaluation will take place on testset (51303 items)
[I 2022-10-11 20:04:20] ...strap/models/factory.py.21: Creating model...
[I 2022-10-11 20:04:22] ...e_networks/networks.py.203: Loading CLIP Visual model...
[I 2022-10-11 20:04:25] ...e_networks/networks.py.145: cuda
[I 2022-10-11 20:04:26] ...networks/tfood_vicha.py.49: Load VCs encoder...
[I 2022-10-11 20:04:28] ...els/metrics/trijoint.py.46: self.save_ids False
[I 2022-10-11 20:04:28] ...strap/models/factory.py.44: Enabling CUDA mode...
[I 2022-10-11 20:04:34] ...etworks/tfood_vicha.py.184: training aux kw modules
[I 2022-10-11 20:04:34] ...etworks/tfood_vicha.py.184: training aux kw modules
[I 2022-10-11 20:04:34] ...otstrap/views/factory.py.8: Creating views...
[I 2022-10-11 20:04:39] ...strap/engines/engine.py.98: Loading last checkpoint
[I 2022-10-11 20:04:39] ...trap/engines/engine.py.440: Loading model...
[I 2022-10-11 20:04:45] ...trap/engines/engine.py.453: Loading engine...
[I 2022-10-11 20:04:45] ...trap/engines/engine.py.407: Loading model... /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_recipe1m_13m_vlp/ckpt_last_model.pth.tar
[I 2022-10-11 20:04:49] ...trap/engines/engine.py.108: Launching evaluation procedures
[I 2022-10-11 20:04:49] ...trap/engines/engine.py.268: Evaluating model on testset for epoch 116
[I 2022-10-11 20:04:49] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler.
[I 2022-10-11 20:04:54] ...trap/engines/engine.py.321: eval: epoch 116 | batch 0/512
[I 2022-10-11 20:04:54] ...trap/engines/engine.py.325:       elapsed: 0:00:05 | left: 0:44:00
[I 2022-10-11 20:04:54] ...trap/engines/engine.py.326:       process: 5.15674 | load: 2.77997
[I 2022-10-11 20:05:07] ...trap/engines/engine.py.321: eval: epoch 116 | batch 10/512
[I 2022-10-11 20:05:07] ...trap/engines/engine.py.325:       elapsed: 0:00:17 | left: 0:14:24
[I 2022-10-11 20:05:07] ...trap/engines/engine.py.326:       process: 1.28958 | load: 0.00019
[I 2022-10-11 20:05:19] ...trap/engines/engine.py.321: eval: epoch 116 | batch 20/512
[I 2022-10-11 20:05:19] ...trap/engines/engine.py.325:       elapsed: 0:00:30 | left: 0:10:47
[I 2022-10-11 20:05:19] ...trap/engines/engine.py.326:       process: 1.27284 | load: 0.00020
[I 2022-10-11 20:05:32] ...trap/engines/engine.py.321: eval: epoch 116 | batch 30/512
[I 2022-10-11 20:05:32] ...trap/engines/engine.py.325:       elapsed: 0:00:43 | left: 0:10:31
[I 2022-10-11 20:05:32] ...trap/engines/engine.py.326:       process: 1.28542 | load: 0.00021
[I 2022-10-11 20:05:46] ...trap/engines/engine.py.321: eval: epoch 116 | batch 40/512
[I 2022-10-11 20:05:46] ...trap/engines/engine.py.325:       elapsed: 0:00:57 | left: 0:10:40
[I 2022-10-11 20:05:46] ...trap/engines/engine.py.326:       process: 1.28462 | load: 0.00019
[I 2022-10-11 20:05:59] ...trap/engines/engine.py.321: eval: epoch 116 | batch 50/512
[I 2022-10-11 20:05:59] ...trap/engines/engine.py.325:       elapsed: 0:01:10 | left: 0:10:00
[I 2022-10-11 20:05:59] ...trap/engines/engine.py.326:       process: 1.35797 | load: 0.00021
[I 2022-10-11 20:06:11] ...trap/engines/engine.py.321: eval: epoch 116 | batch 60/512
[I 2022-10-11 20:06:11] ...trap/engines/engine.py.325:       elapsed: 0:01:22 | left: 0:09:33
[I 2022-10-11 20:06:11] ...trap/engines/engine.py.326:       process: 1.20689 | load: 0.00021
[I 2022-10-11 20:06:24] ...trap/engines/engine.py.321: eval: epoch 116 | batch 70/512
[I 2022-10-11 20:06:24] ...trap/engines/engine.py.325:       elapsed: 0:01:35 | left: 0:09:18
[I 2022-10-11 20:06:24] ...trap/engines/engine.py.326:       process: 1.36269 | load: 0.00020
[I 2022-10-11 20:06:36] ...trap/engines/engine.py.321: eval: epoch 116 | batch 80/512
[I 2022-10-11 20:06:36] ...trap/engines/engine.py.325:       elapsed: 0:01:47 | left: 0:08:36
[I 2022-10-11 20:06:36] ...trap/engines/engine.py.326:       process: 1.17003 | load: 0.00019
[I 2022-10-11 20:06:48] ...trap/engines/engine.py.321: eval: epoch 116 | batch 90/512
[I 2022-10-11 20:06:48] ...trap/engines/engine.py.325:       elapsed: 0:01:59 | left: 0:08:24
[I 2022-10-11 20:06:48] ...trap/engines/engine.py.326:       process: 1.19402 | load: 0.00020
[I 2022-10-11 20:07:00] ...trap/engines/engine.py.321: eval: epoch 116 | batch 100/512
[I 2022-10-11 20:07:00] ...trap/engines/engine.py.325:       elapsed: 0:02:11 | left: 0:08:27
[I 2022-10-11 20:07:00] ...trap/engines/engine.py.326:       process: 1.20384 | load: 0.00021
[I 2022-10-11 20:07:47] ...ytorch/bootstrap/run.py.73: Saving environment info
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75: options: 
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:   dataset:
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     aux_kw_path:
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       test: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_test_titles_kw.json
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       train: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_train_titles_kw.json
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       val: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_val_titles_kw.json
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     batch_sampler: triplet_classif
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     batch_size: 100
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     database: lmdb
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     dir: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     eval_split: test
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     freq_mismatch: 0.0
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     image_from: database
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     import: recipe1m.datasets.factory
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     kw_path:
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       test: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_test_ingr_kw.json
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       train: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_train_ingr_kw.json
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       val: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_val_ingr_kw.json
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     name: recipe1m
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     nb_classes: 1048
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     nb_threads: 4
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     randkw_p: 0.3
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     randkw_p_aux: 0.5
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     tokenized_raw_text: True
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     train_split: None
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     use_vcs: True
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:   engine:
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     name: logger
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     nb_epochs: 120
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     print_freq: 10
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     saving_criteria: ['eval_epoch.metric.med_im2recipe_mean:min', 'eval_epoch.metric.recall_at_1_im2recipe_mean:max']
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:   exp:
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     checkpoint: None
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     dir: /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     resume: last
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:   misc:
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     cuda: True
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     data_parrallel: True
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     logs_name: None
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     overrite: False
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     seed: 1338
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:   model:
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     criterion:
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:         tri_cross_encoder: False
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       itm_loss_weight: 1
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       keep_background: False
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       retrieval_strategy:
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:         aggregation: valid
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:         get_tokens: True
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:         margin: 0.05
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:         margin_params:
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:           increment: 0.005
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:           increment_margin: True
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:           max_margin: 0.3
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:         name: triplet
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:         nb_samples: 9999
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:         sampling: max_negative
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:         substrategy: ['IRR', 'RII', 'SIRR', 'SRII']
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:         substrategy_weights: [1.0, 1.0, 0.1, 0.1]
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     import: recipe1m.models.factory
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     metric:
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       k_test: 10
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       nb_bags: 10
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       nb_matchs_per_bag: 1000
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       retrieval_dir: /gpfsscratch/rech/dyf/ugz83ue/tmp/recipe1m
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       trijoint: True
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     name: trijoint
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     network:
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       activations: ['tanh', 'normalize']
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       aux_kwords: True
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       aux_kwords_encoder: True
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       avg_concat: True
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       bert_config: /linkhome/rech/genisi01/ugz83ue/tfood/recipe1m/models/networks/recipe_networks/config_bert_albef.json
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       cat_pos: 0
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       checkpoint: None
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       class_attention: False
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       class_attention_fus: False
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       cls_token: False
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       cls_token_fus: False
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       cross_decoder_image: True
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       cross_decoder_recipe: True
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:         class_attention: False
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:         cls_token: False
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:         cross_attention: False
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:         cross_decoder: True
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:         double_decoder: False
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:         get_tokens: False
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:         load_backbones: False
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:         n_heads: 4
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:         n_layers: 4
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:         query_img: False
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       cross_transformer: False
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       dim_emb: 1024
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       dim_image_out: 2048
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       dim_ingr_out: 300
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       dim_instr_in: 1024
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       dim_instr_out: 1024
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       embedding_names: ['proj_image']
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       freeze_im: False
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       freeze_rec: False
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       get_tokens: True
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       get_tokens_cross_decoder_recipe: True
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       hidden_size: 512
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       image_backbone_name: clip
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       image_res: 224
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       kwords_same_level: False
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       last_cat_aux: True
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       n_heads: 4
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       n_heads_cross: 4
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       n_heads_cross_image: 4
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       n_heads_single: 4
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       n_layers: 2
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       n_layers_cross: 2
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       n_layers_cross_image: 1
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       n_layers_single: 2
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       num_hidden_layers_kw: 2
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       old: True
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       path_ingrs: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/text/vocab.pkl
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       path_vocab: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/text/vocab_all.txt
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       recipe_encoder: tfoodvicha-transformer
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       text_encoder: bert-base-uncased
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       vision_width: 768
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       vit_all_tokens: True
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       with_ingrs: True
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       with_instrs: True
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:       with_titles: True
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     with_classif: False
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:   optimizer:
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     batch_size_factor: None
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     clip_grad: 8.0
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     import: recipe1m.optimizers.factory
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     lr: 1e-05
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     lr_img: 1e-06
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     name: trijoint_fixed_fine_tune
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:     switch_epoch: 20
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:   path_opts: /linkhome/rech/genisi01/ugz83ue/tfood/recipe1m/options/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_noimgproj_keepall.yaml
[S 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.75:   view: ['logs:train_epoch.loss', 'logs:train_epoch.bad_pairs', 'logs:eval_epoch.metric.med_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_1_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_5_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_10_im2recipe_mean', 'logs:eval_epoch.metric.med_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_1_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_5_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_10_recipe2im_mean', 'logs:optimizer.is_optimizer_recipe&image', 'logs:optimizer.total_norm']
[I 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.76: posix.uname_result(sysname='Linux', nodename='jean-zay-ia825', release='4.18.0-305.57.1.el8_4.x86_64', version='#1 SMP Tue Jul 5 17:55:55 EDT 2022', machine='x86_64')
[I 2022-10-11 20:07:49] ...ytorch/bootstrap/run.py.80: Available GPUs: [0]
[I 2022-10-11 20:07:49] ...strap/engines/factory.py.9: Creating engine...
[I 2022-10-11 20:07:49] ...trap/datasets/factory.py.8: Creating dataset...
[I 2022-10-11 20:07:53] ...m/datasets/recipe1m.py.152: Load VCs...
[I 2022-10-11 20:07:54] ...m/datasets/recipe1m.py.165: randkw_p... None
[I 2022-10-11 20:08:59] ...ytorch/bootstrap/run.py.73: Saving environment info
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75: options: 
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:   dataset:
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     aux_kw_path:
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       test: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_test_titles_kw.json
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       train: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_train_titles_kw.json
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       val: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_val_titles_kw.json
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     batch_sampler: triplet_classif
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     batch_size: 100
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     database: lmdb
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     dir: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     eval_split: test
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     freq_mismatch: 0.0
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     image_from: database
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     import: recipe1m.datasets.factory
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     kw_path:
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       test: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_test_ingr_kw.json
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       train: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_train_ingr_kw.json
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       val: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_val_ingr_kw.json
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     name: recipe1m
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     nb_classes: 1048
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     nb_threads: 4
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     randkw_p: 0.3
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     randkw_p_aux: 0.5
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     tokenized_raw_text: True
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     train_split: None
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     use_vcs: True
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:   engine:
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     name: logger
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     nb_epochs: 120
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     print_freq: 10
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     saving_criteria: ['eval_epoch.metric.med_im2recipe_mean:min', 'eval_epoch.metric.recall_at_1_im2recipe_mean:max']
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:   exp:
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     checkpoint: None
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     dir: /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     resume: best_eval_epoch.metric.recall_at_1_im2recipe_mean
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:   misc:
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     cuda: True
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     data_parrallel: True
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     logs_name: None
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     overrite: False
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     seed: 1338
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:   model:
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     criterion:
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:         tri_cross_encoder: False
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       itm_loss_weight: 1
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       keep_background: False
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       retrieval_strategy:
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:         aggregation: valid
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:         get_tokens: True
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:         margin: 0.05
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:         margin_params:
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:           increment: 0.005
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:           increment_margin: True
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:           max_margin: 0.3
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:         name: triplet
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:         nb_samples: 9999
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:         sampling: max_negative
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:         substrategy: ['IRR', 'RII', 'SIRR', 'SRII']
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:         substrategy_weights: [1.0, 1.0, 0.1, 0.1]
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     import: recipe1m.models.factory
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     metric:
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       k_test: 10
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       nb_bags: 10
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       nb_matchs_per_bag: 1000
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       retrieval_dir: /gpfsscratch/rech/dyf/ugz83ue/tmp/recipe1m
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       trijoint: True
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     name: trijoint
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     network:
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       activations: ['tanh', 'normalize']
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       aux_kwords: True
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       aux_kwords_encoder: True
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       avg_concat: True
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       bert_config: /linkhome/rech/genisi01/ugz83ue/tfood/recipe1m/models/networks/recipe_networks/config_bert_albef.json
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       cat_pos: 0
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       checkpoint: None
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       class_attention: False
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       class_attention_fus: False
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       cls_token: False
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       cls_token_fus: False
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       cross_decoder_image: True
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       cross_decoder_recipe: True
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:         class_attention: False
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:         cls_token: False
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:         cross_attention: False
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:         cross_decoder: True
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:         double_decoder: False
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:         get_tokens: False
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:         load_backbones: False
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:         n_heads: 4
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:         n_layers: 4
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:         query_img: False
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       cross_transformer: False
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       dim_emb: 1024
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       dim_image_out: 2048
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       dim_ingr_out: 300
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       dim_instr_in: 1024
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       dim_instr_out: 1024
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       embedding_names: ['proj_image']
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       freeze_im: False
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       freeze_rec: False
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       get_tokens: True
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       get_tokens_cross_decoder_recipe: True
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       hidden_size: 512
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       image_backbone_name: clip
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       image_res: 224
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       kwords_same_level: False
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       last_cat_aux: True
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       n_heads: 4
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       n_heads_cross: 4
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       n_heads_cross_image: 4
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       n_heads_single: 4
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       n_layers: 2
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       n_layers_cross: 2
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       n_layers_cross_image: 1
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       n_layers_single: 2
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       num_hidden_layers_kw: 2
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       old: True
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       path_ingrs: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/text/vocab.pkl
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       path_vocab: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/text/vocab_all.txt
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       recipe_encoder: tfoodvicha-transformer
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       text_encoder: bert-base-uncased
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       vision_width: 768
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       vit_all_tokens: True
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       with_ingrs: True
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       with_instrs: True
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:       with_titles: True
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     with_classif: False
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:   optimizer:
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     batch_size_factor: None
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     clip_grad: 8.0
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     import: recipe1m.optimizers.factory
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     lr: 1e-05
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     lr_img: 1e-06
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     name: trijoint_fixed_fine_tune
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:     switch_epoch: 20
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:   path_opts: /linkhome/rech/genisi01/ugz83ue/tfood/recipe1m/options/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_noimgproj_keepall.yaml
[S 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.75:   view: ['logs:train_epoch.loss', 'logs:train_epoch.bad_pairs', 'logs:eval_epoch.metric.med_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_1_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_5_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_10_im2recipe_mean', 'logs:eval_epoch.metric.med_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_1_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_5_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_10_recipe2im_mean', 'logs:optimizer.is_optimizer_recipe&image', 'logs:optimizer.total_norm']
[I 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.76: posix.uname_result(sysname='Linux', nodename='jean-zay-ia825', release='4.18.0-305.57.1.el8_4.x86_64', version='#1 SMP Tue Jul 5 17:55:55 EDT 2022', machine='x86_64')
[I 2022-10-11 20:09:00] ...ytorch/bootstrap/run.py.80: Available GPUs: [0]
[I 2022-10-11 20:09:00] ...strap/engines/factory.py.9: Creating engine...
[I 2022-10-11 20:09:00] ...trap/datasets/factory.py.8: Creating dataset...
[I 2022-10-11 20:09:04] ...m/datasets/recipe1m.py.152: Load VCs...
[I 2022-10-11 20:09:05] ...m/datasets/recipe1m.py.165: randkw_p... None
[I 2022-10-11 20:10:05] ...rap/datasets/factory.py.21: Evaluation will take place on testset (51303 items)
[I 2022-10-11 20:10:05] ...strap/models/factory.py.21: Creating model...
[I 2022-10-11 20:10:07] ...e_networks/networks.py.203: Loading CLIP Visual model...
[I 2022-10-11 20:10:10] ...e_networks/networks.py.145: cuda
[I 2022-10-11 20:10:11] ...networks/tfood_vicha.py.49: Load VCs encoder...
[I 2022-10-11 20:10:13] ...els/metrics/trijoint.py.46: self.save_ids False
[I 2022-10-11 20:10:13] ...strap/models/factory.py.44: Enabling CUDA mode...
[I 2022-10-11 20:10:15] ...etworks/tfood_vicha.py.184: training aux kw modules
[I 2022-10-11 20:10:15] ...etworks/tfood_vicha.py.184: training aux kw modules
[I 2022-10-11 20:10:15] ...otstrap/views/factory.py.8: Creating views...
[I 2022-10-11 20:10:16] ...strap/engines/engine.py.98: Loading best_eval_epoch.metric.recall_at_1_im2recipe_mean checkpoint
[I 2022-10-11 20:10:16] ...trap/engines/engine.py.440: Loading model...
[I 2022-10-11 20:10:23] ...trap/engines/engine.py.453: Loading engine...
[I 2022-10-11 20:10:23] ...trap/engines/engine.py.108: Launching evaluation procedures
[I 2022-10-11 20:10:23] ...trap/engines/engine.py.268: Evaluating model on testset for epoch 110
[I 2022-10-11 20:10:23] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler.
[I 2022-10-11 20:10:28] ...trap/engines/engine.py.321: eval: epoch 110 | batch 0/512
[I 2022-10-11 20:10:28] ...trap/engines/engine.py.325:       elapsed: 0:00:04 | left: 0:39:25
[I 2022-10-11 20:10:28] ...trap/engines/engine.py.326:       process: 4.61958 | load: 2.71153
[I 2022-10-11 20:10:40] ...trap/engines/engine.py.321: eval: epoch 110 | batch 10/512
[I 2022-10-11 20:10:40] ...trap/engines/engine.py.325:       elapsed: 0:00:16 | left: 0:13:31
[I 2022-10-11 20:10:40] ...trap/engines/engine.py.326:       process: 1.24060 | load: 0.00016
[I 2022-10-11 20:10:53] ...trap/engines/engine.py.321: eval: epoch 110 | batch 20/512
[I 2022-10-11 20:10:53] ...trap/engines/engine.py.325:       elapsed: 0:00:29 | left: 0:10:37
[I 2022-10-11 20:10:53] ...trap/engines/engine.py.326:       process: 1.22213 | load: 0.00020
[I 2022-10-11 20:11:05] ...trap/engines/engine.py.321: eval: epoch 110 | batch 30/512
[I 2022-10-11 20:11:05] ...trap/engines/engine.py.325:       elapsed: 0:00:41 | left: 0:10:01
[I 2022-10-11 20:11:05] ...trap/engines/engine.py.326:       process: 1.26941 | load: 0.00017
[I 2022-10-11 20:11:17] ...trap/engines/engine.py.321: eval: epoch 110 | batch 40/512
[I 2022-10-11 20:11:17] ...trap/engines/engine.py.325:       elapsed: 0:00:54 | left: 0:09:42
[I 2022-10-11 20:11:17] ...trap/engines/engine.py.326:       process: 1.18267 | load: 0.00017
[I 2022-10-11 20:11:30] ...trap/engines/engine.py.321: eval: epoch 110 | batch 50/512
[I 2022-10-11 20:11:30] ...trap/engines/engine.py.325:       elapsed: 0:01:06 | left: 0:09:28
[I 2022-10-11 20:11:30] ...trap/engines/engine.py.326:       process: 1.22295 | load: 0.00020
[I 2022-10-11 20:11:42] ...trap/engines/engine.py.321: eval: epoch 110 | batch 60/512
[I 2022-10-11 20:11:42] ...trap/engines/engine.py.325:       elapsed: 0:01:18 | left: 0:09:04
[I 2022-10-11 20:11:42] ...trap/engines/engine.py.326:       process: 1.19185 | load: 0.00021
[I 2022-10-11 20:11:54] ...trap/engines/engine.py.321: eval: epoch 110 | batch 70/512
[I 2022-10-11 20:11:54] ...trap/engines/engine.py.325:       elapsed: 0:01:30 | left: 0:08:53
[I 2022-10-11 20:11:54] ...trap/engines/engine.py.326:       process: 1.20593 | load: 0.00021
[I 2022-10-11 20:12:06] ...trap/engines/engine.py.321: eval: epoch 110 | batch 80/512
[I 2022-10-11 20:12:06] ...trap/engines/engine.py.325:       elapsed: 0:01:42 | left: 0:08:35
[I 2022-10-11 20:12:06] ...trap/engines/engine.py.326:       process: 1.19562 | load: 0.00017
[I 2022-10-11 20:12:18] ...trap/engines/engine.py.321: eval: epoch 110 | batch 90/512
[I 2022-10-11 20:12:18] ...trap/engines/engine.py.325:       elapsed: 0:01:54 | left: 0:08:28
[I 2022-10-11 20:12:18] ...trap/engines/engine.py.326:       process: 1.21391 | load: 0.00017
[I 2022-10-11 20:12:30] ...trap/engines/engine.py.321: eval: epoch 110 | batch 100/512
[I 2022-10-11 20:12:30] ...trap/engines/engine.py.325:       elapsed: 0:02:06 | left: 0:08:20
[I 2022-10-11 20:12:30] ...trap/engines/engine.py.326:       process: 1.08455 | load: 0.00017
[I 2022-10-11 20:12:41] ...trap/engines/engine.py.321: eval: epoch 110 | batch 110/512
[I 2022-10-11 20:12:41] ...trap/engines/engine.py.325:       elapsed: 0:02:17 | left: 0:07:19
[I 2022-10-11 20:12:41] ...trap/engines/engine.py.326:       process: 1.07935 | load: 0.00016
[I 2022-10-11 20:12:52] ...trap/engines/engine.py.321: eval: epoch 110 | batch 120/512
[I 2022-10-11 20:12:52] ...trap/engines/engine.py.325:       elapsed: 0:02:28 | left: 0:07:04
[I 2022-10-11 20:12:52] ...trap/engines/engine.py.326:       process: 1.07527 | load: 0.00018
[I 2022-10-11 20:13:02] ...trap/engines/engine.py.321: eval: epoch 110 | batch 130/512
[I 2022-10-11 20:13:02] ...trap/engines/engine.py.325:       elapsed: 0:02:39 | left: 0:06:53
[I 2022-10-11 20:13:02] ...trap/engines/engine.py.326:       process: 1.08292 | load: 0.00014
[I 2022-10-11 20:13:13] ...trap/engines/engine.py.321: eval: epoch 110 | batch 140/512
[I 2022-10-11 20:13:13] ...trap/engines/engine.py.325:       elapsed: 0:02:50 | left: 0:06:40
[I 2022-10-11 20:13:13] ...trap/engines/engine.py.326:       process: 1.07985 | load: 0.00016
[I 2022-10-11 20:13:24] ...trap/engines/engine.py.321: eval: epoch 110 | batch 150/512
[I 2022-10-11 20:13:24] ...trap/engines/engine.py.325:       elapsed: 0:03:00 | left: 0:06:29
[I 2022-10-11 20:13:24] ...trap/engines/engine.py.326:       process: 1.06671 | load: 0.00016
[I 2022-10-11 20:13:35] ...trap/engines/engine.py.321: eval: epoch 110 | batch 160/512
[I 2022-10-11 20:13:35] ...trap/engines/engine.py.325:       elapsed: 0:03:11 | left: 0:06:19
[I 2022-10-11 20:13:35] ...trap/engines/engine.py.326:       process: 1.05490 | load: 0.00016
[I 2022-10-11 20:13:46] ...trap/engines/engine.py.321: eval: epoch 110 | batch 170/512
[I 2022-10-11 20:13:46] ...trap/engines/engine.py.325:       elapsed: 0:03:22 | left: 0:06:10
[I 2022-10-11 20:13:46] ...trap/engines/engine.py.326:       process: 1.09118 | load: 0.00016
[I 2022-10-11 20:13:57] ...trap/engines/engine.py.321: eval: epoch 110 | batch 180/512
[I 2022-10-11 20:13:57] ...trap/engines/engine.py.325:       elapsed: 0:03:33 | left: 0:06:01
[I 2022-10-11 20:13:57] ...trap/engines/engine.py.326:       process: 1.08942 | load: 0.00017
[I 2022-10-11 20:14:07] ...trap/engines/engine.py.321: eval: epoch 110 | batch 190/512
[I 2022-10-11 20:14:07] ...trap/engines/engine.py.325:       elapsed: 0:03:44 | left: 0:05:49
[I 2022-10-11 20:14:07] ...trap/engines/engine.py.326:       process: 1.10249 | load: 0.00017
[I 2022-10-11 20:14:18] ...trap/engines/engine.py.321: eval: epoch 110 | batch 200/512
[I 2022-10-11 20:14:18] ...trap/engines/engine.py.325:       elapsed: 0:03:55 | left: 0:05:39
[I 2022-10-11 20:14:18] ...trap/engines/engine.py.326:       process: 1.08840 | load: 0.00018
[I 2022-10-11 20:14:29] ...trap/engines/engine.py.321: eval: epoch 110 | batch 210/512
[I 2022-10-11 20:14:29] ...trap/engines/engine.py.325:       elapsed: 0:04:05 | left: 0:05:26
[I 2022-10-11 20:14:29] ...trap/engines/engine.py.326:       process: 1.08658 | load: 0.00022
[I 2022-10-11 20:14:40] ...trap/engines/engine.py.321: eval: epoch 110 | batch 220/512
[I 2022-10-11 20:14:40] ...trap/engines/engine.py.325:       elapsed: 0:04:16 | left: 0:05:17
[I 2022-10-11 20:14:40] ...trap/engines/engine.py.326:       process: 1.09008 | load: 0.00021
[I 2022-10-11 20:14:51] ...trap/engines/engine.py.321: eval: epoch 110 | batch 230/512
[I 2022-10-11 20:14:51] ...trap/engines/engine.py.325:       elapsed: 0:04:27 | left: 0:05:04
[I 2022-10-11 20:14:51] ...trap/engines/engine.py.326:       process: 1.06579 | load: 0.00022
[I 2022-10-11 20:15:02] ...trap/engines/engine.py.321: eval: epoch 110 | batch 240/512
[I 2022-10-11 20:15:02] ...trap/engines/engine.py.325:       elapsed: 0:04:38 | left: 0:04:55
[I 2022-10-11 20:15:02] ...trap/engines/engine.py.326:       process: 1.08816 | load: 0.00016
[I 2022-10-11 20:15:13] ...trap/engines/engine.py.321: eval: epoch 110 | batch 250/512
[I 2022-10-11 20:15:13] ...trap/engines/engine.py.325:       elapsed: 0:04:49 | left: 0:04:45
[I 2022-10-11 20:15:13] ...trap/engines/engine.py.326:       process: 1.08744 | load: 0.00017
[I 2022-10-11 20:15:23] ...trap/engines/engine.py.321: eval: epoch 110 | batch 260/512
[I 2022-10-11 20:15:23] ...trap/engines/engine.py.325:       elapsed: 0:05:00 | left: 0:04:33
[I 2022-10-11 20:15:23] ...trap/engines/engine.py.326:       process: 1.09358 | load: 0.00017
[I 2022-10-11 20:15:34] ...trap/engines/engine.py.321: eval: epoch 110 | batch 270/512
[I 2022-10-11 20:15:34] ...trap/engines/engine.py.325:       elapsed: 0:05:11 | left: 0:04:23
[I 2022-10-11 20:15:34] ...trap/engines/engine.py.326:       process: 1.09453 | load: 0.00017
[I 2022-10-11 20:15:45] ...trap/engines/engine.py.321: eval: epoch 110 | batch 280/512
[I 2022-10-11 20:15:45] ...trap/engines/engine.py.325:       elapsed: 0:05:21 | left: 0:04:10
[I 2022-10-11 20:15:45] ...trap/engines/engine.py.326:       process: 1.07462 | load: 0.00017
[I 2022-10-11 20:15:56] ...trap/engines/engine.py.321: eval: epoch 110 | batch 290/512
[I 2022-10-11 20:15:56] ...trap/engines/engine.py.325:       elapsed: 0:05:32 | left: 0:04:00
[I 2022-10-11 20:15:56] ...trap/engines/engine.py.326:       process: 1.07512 | load: 0.00017
[I 2022-10-11 20:16:07] ...trap/engines/engine.py.321: eval: epoch 110 | batch 300/512
[I 2022-10-11 20:16:07] ...trap/engines/engine.py.325:       elapsed: 0:05:43 | left: 0:03:49
[I 2022-10-11 20:16:07] ...trap/engines/engine.py.326:       process: 1.08476 | load: 0.00022
[I 2022-10-11 20:16:18] ...trap/engines/engine.py.321: eval: epoch 110 | batch 310/512
[I 2022-10-11 20:16:18] ...trap/engines/engine.py.325:       elapsed: 0:05:54 | left: 0:03:40
[I 2022-10-11 20:16:18] ...trap/engines/engine.py.326:       process: 1.08567 | load: 0.00021
[I 2022-10-11 20:16:29] ...trap/engines/engine.py.321: eval: epoch 110 | batch 320/512
[I 2022-10-11 20:16:29] ...trap/engines/engine.py.325:       elapsed: 0:06:05 | left: 0:03:29
[I 2022-10-11 20:16:29] ...trap/engines/engine.py.326:       process: 1.09809 | load: 0.00021
[I 2022-10-11 20:16:39] ...trap/engines/engine.py.321: eval: epoch 110 | batch 330/512
[I 2022-10-11 20:16:39] ...trap/engines/engine.py.325:       elapsed: 0:06:16 | left: 0:03:17
[I 2022-10-11 20:16:39] ...trap/engines/engine.py.326:       process: 1.07298 | load: 0.00023
[I 2022-10-11 20:16:50] ...trap/engines/engine.py.321: eval: epoch 110 | batch 340/512
[I 2022-10-11 20:16:50] ...trap/engines/engine.py.325:       elapsed: 0:06:26 | left: 0:03:04
[I 2022-10-11 20:16:50] ...trap/engines/engine.py.326:       process: 1.05607 | load: 0.00019
[I 2022-10-11 20:17:01] ...trap/engines/engine.py.321: eval: epoch 110 | batch 350/512
[I 2022-10-11 20:17:01] ...trap/engines/engine.py.325:       elapsed: 0:06:37 | left: 0:02:55
[I 2022-10-11 20:17:01] ...trap/engines/engine.py.326:       process: 1.09391 | load: 0.00023
[I 2022-10-11 20:17:12] ...trap/engines/engine.py.321: eval: epoch 110 | batch 360/512
[I 2022-10-11 20:17:12] ...trap/engines/engine.py.325:       elapsed: 0:06:48 | left: 0:02:44
[I 2022-10-11 20:17:12] ...trap/engines/engine.py.326:       process: 1.09614 | load: 0.00021
[I 2022-10-11 20:17:23] ...trap/engines/engine.py.321: eval: epoch 110 | batch 370/512
[I 2022-10-11 20:17:23] ...trap/engines/engine.py.325:       elapsed: 0:06:59 | left: 0:02:34
[I 2022-10-11 20:17:23] ...trap/engines/engine.py.326:       process: 1.09172 | load: 0.00021
[I 2022-10-11 20:17:34] ...trap/engines/engine.py.321: eval: epoch 110 | batch 380/512
[I 2022-10-11 20:17:34] ...trap/engines/engine.py.325:       elapsed: 0:07:10 | left: 0:02:24
[I 2022-10-11 20:17:34] ...trap/engines/engine.py.326:       process: 1.09825 | load: 0.00023
[I 2022-10-11 20:17:44] ...trap/engines/engine.py.321: eval: epoch 110 | batch 390/512
[I 2022-10-11 20:17:44] ...trap/engines/engine.py.325:       elapsed: 0:07:21 | left: 0:02:11
[I 2022-10-11 20:17:44] ...trap/engines/engine.py.326:       process: 1.07129 | load: 0.00021
[I 2022-10-11 20:17:55] ...trap/engines/engine.py.321: eval: epoch 110 | batch 400/512
[I 2022-10-11 20:17:55] ...trap/engines/engine.py.325:       elapsed: 0:07:32 | left: 0:02:01
[I 2022-10-11 20:17:55] ...trap/engines/engine.py.326:       process: 1.09616 | load: 0.00023
[I 2022-10-11 20:18:06] ...trap/engines/engine.py.321: eval: epoch 110 | batch 410/512
[I 2022-10-11 20:18:06] ...trap/engines/engine.py.325:       elapsed: 0:07:42 | left: 0:01:50
[I 2022-10-11 20:18:06] ...trap/engines/engine.py.326:       process: 1.07471 | load: 0.00022
[I 2022-10-11 20:18:17] ...trap/engines/engine.py.321: eval: epoch 110 | batch 420/512
[I 2022-10-11 20:18:17] ...trap/engines/engine.py.325:       elapsed: 0:07:53 | left: 0:01:38
[I 2022-10-11 20:18:17] ...trap/engines/engine.py.326:       process: 1.07567 | load: 0.00021
[I 2022-10-11 20:18:28] ...trap/engines/engine.py.321: eval: epoch 110 | batch 430/512
[I 2022-10-11 20:18:28] ...trap/engines/engine.py.325:       elapsed: 0:08:04 | left: 0:01:29
[I 2022-10-11 20:18:28] ...trap/engines/engine.py.326:       process: 1.09650 | load: 0.00021
[I 2022-10-11 20:18:39] ...trap/engines/engine.py.321: eval: epoch 110 | batch 440/512
[I 2022-10-11 20:18:39] ...trap/engines/engine.py.325:       elapsed: 0:08:15 | left: 0:01:17
[I 2022-10-11 20:18:39] ...trap/engines/engine.py.326:       process: 1.09587 | load: 0.00023
[I 2022-10-11 20:18:49] ...trap/engines/engine.py.321: eval: epoch 110 | batch 450/512
[I 2022-10-11 20:18:49] ...trap/engines/engine.py.325:       elapsed: 0:08:26 | left: 0:01:07
[I 2022-10-11 20:18:49] ...trap/engines/engine.py.326:       process: 1.10550 | load: 0.00021
[I 2022-10-11 20:19:00] ...trap/engines/engine.py.321: eval: epoch 110 | batch 460/512
[I 2022-10-11 20:19:00] ...trap/engines/engine.py.325:       elapsed: 0:08:37 | left: 0:00:56
[I 2022-10-11 20:19:00] ...trap/engines/engine.py.326:       process: 1.09971 | load: 0.00022
[I 2022-10-11 20:19:11] ...trap/engines/engine.py.321: eval: epoch 110 | batch 470/512
[I 2022-10-11 20:19:11] ...trap/engines/engine.py.325:       elapsed: 0:08:48 | left: 0:00:45
[I 2022-10-11 20:19:11] ...trap/engines/engine.py.326:       process: 1.09228 | load: 0.00021
[I 2022-10-11 20:19:22] ...trap/engines/engine.py.321: eval: epoch 110 | batch 480/512
[I 2022-10-11 20:19:22] ...trap/engines/engine.py.325:       elapsed: 0:08:58 | left: 0:00:34
[I 2022-10-11 20:19:22] ...trap/engines/engine.py.326:       process: 1.09342 | load: 0.00021
[I 2022-10-11 20:19:33] ...trap/engines/engine.py.321: eval: epoch 110 | batch 490/512
[I 2022-10-11 20:19:33] ...trap/engines/engine.py.325:       elapsed: 0:09:09 | left: 0:00:23
[I 2022-10-11 20:19:33] ...trap/engines/engine.py.326:       process: 1.05746 | load: 0.00021
[I 2022-10-11 20:19:44] ...trap/engines/engine.py.321: eval: epoch 110 | batch 500/512
[I 2022-10-11 20:19:44] ...trap/engines/engine.py.325:       elapsed: 0:09:20 | left: 0:00:13
[I 2022-10-11 20:19:44] ...trap/engines/engine.py.326:       process: 1.08447 | load: 0.00021
[I 2022-10-11 20:19:55] ...trap/engines/engine.py.321: eval: epoch 110 | batch 510/512
[I 2022-10-11 20:19:55] ...trap/engines/engine.py.325:       elapsed: 0:09:31 | left: 0:00:02
[I 2022-10-11 20:19:55] ...trap/engines/engine.py.326:       process: 1.08495 | load: 0.00017
[S 2022-10-11 20:19:57] ...trap/engines/engine.py.113: eval_epoch.epoch: 110
[I 2022-10-11 20:19:57] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs
[S 2022-10-11 20:20:23] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00
[S 2022-10-11 20:20:23] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00
[S 2022-10-11 20:20:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.740600
[S 2022-10-11 20:20:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.913600
[S 2022-10-11 20:20:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.935000
[S 2022-10-11 20:20:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.742900
[S 2022-10-11 20:20:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.911300
[S 2022-10-11 20:20:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.934800
[S 2022-10-11 20:20:23] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000
[S 2022-10-11 20:20:23] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000
[S 2022-10-11 20:20:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.010151
[S 2022-10-11 20:20:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.006844
[S 2022-10-11 20:20:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.009230
[S 2022-10-11 20:20:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.007148
[S 2022-10-11 20:20:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.007349
[S 2022-10-11 20:20:23] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.007574
[I 2022-10-11 20:20:36] ...trap/engines/engine.py.115: Ending evaluation procedures
[W 2022-10-11 20:20:39] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json'
[W 2022-10-11 20:20:39] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[W 2022-10-11 20:20:39] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[I 2022-10-11 20:20:47] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html
[I 2022-10-11 21:51:56] ...ytorch/bootstrap/run.py.73: Saving environment info
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75: options: 
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:   dataset:
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     aux_kw_path:
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       test: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_test_titles_kw.json
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       train: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_train_titles_kw.json
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       val: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_val_titles_kw.json
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     batch_sampler: triplet_classif
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     batch_size: 100
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     database: lmdb
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     dir: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     eval_split: test
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     freq_mismatch: 0.0
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     image_from: database
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     import: recipe1m.datasets.factory
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     kw_path:
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       test: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_test_ingr_kw.json
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       train: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_train_ingr_kw.json
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       val: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_val_ingr_kw.json
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     name: recipe1m
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     nb_classes: 1048
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     nb_threads: 4
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     randkw_p: 0.3
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     randkw_p_aux: 0.5
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     tokenized_raw_text: True
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     train_split: None
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     use_vcs: True
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:   engine:
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     name: logger
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     nb_epochs: 120
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     print_freq: 10
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     saving_criteria: ['eval_epoch.metric.med_im2recipe_mean:min', 'eval_epoch.metric.recall_at_1_im2recipe_mean:max']
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:   exp:
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     checkpoint: None
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     dir: /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     resume: best_eval_epoch.metric.recall_at_1_im2recipe_mean
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:   misc:
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     cuda: True
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     data_parrallel: True
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     logs_name: None
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     overrite: False
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     seed: 1338
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:   model:
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     criterion:
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:         tri_cross_encoder: False
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       itm_loss_weight: 1
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       keep_background: False
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       retrieval_strategy:
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:         aggregation: valid
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:         get_tokens: True
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:         margin: 0.05
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:         margin_params:
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:           increment: 0.005
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:           increment_margin: True
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:           max_margin: 0.3
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:         name: triplet
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:         nb_samples: 9999
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:         sampling: max_negative
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:         substrategy: ['IRR', 'RII', 'SIRR', 'SRII']
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:         substrategy_weights: [1.0, 1.0, 0.1, 0.1]
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     import: recipe1m.models.factory
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     metric:
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       k_test: 10
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       nb_bags: 10
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       nb_matchs_per_bag: 1000
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       retrieval_dir: /gpfsscratch/rech/dyf/ugz83ue/tmp/recipe1m
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       trijoint: True
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     name: trijoint
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     network:
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       activations: ['tanh', 'normalize']
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       aux_kwords: True
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       aux_kwords_encoder: True
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       avg_concat: True
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       bert_config: /linkhome/rech/genisi01/ugz83ue/tfood/recipe1m/models/networks/recipe_networks/config_bert_albef.json
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       cat_pos: 0
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       checkpoint: None
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       class_attention: False
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       class_attention_fus: False
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       cls_token: False
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       cls_token_fus: False
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       cross_decoder_image: True
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       cross_decoder_recipe: True
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:         class_attention: False
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:         cls_token: False
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:         cross_attention: False
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:         cross_decoder: True
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:         double_decoder: False
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:         get_tokens: False
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:         load_backbones: False
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:         n_heads: 4
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:         n_layers: 4
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:         query_img: False
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       cross_transformer: False
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       dim_emb: 1024
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       dim_image_out: 2048
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       dim_ingr_out: 300
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       dim_instr_in: 1024
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       dim_instr_out: 1024
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       embedding_names: ['proj_image']
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       freeze_im: False
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       freeze_rec: False
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       get_tokens: True
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       get_tokens_cross_decoder_recipe: True
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       hidden_size: 512
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       image_backbone_name: clip
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       image_res: 224
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       kwords_same_level: False
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       last_cat_aux: True
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       n_heads: 4
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       n_heads_cross: 4
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       n_heads_cross_image: 4
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       n_heads_single: 4
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       n_layers: 2
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       n_layers_cross: 2
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       n_layers_cross_image: 1
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       n_layers_single: 2
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       num_hidden_layers_kw: 2
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       old: True
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       path_ingrs: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/text/vocab.pkl
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       path_vocab: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/text/vocab_all.txt
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       recipe_encoder: tfoodvicha-transformer
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       text_encoder: bert-base-uncased
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       vision_width: 768
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       vit_all_tokens: True
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       with_ingrs: True
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       with_instrs: True
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:       with_titles: True
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     with_classif: False
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:   optimizer:
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     batch_size_factor: None
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     clip_grad: 8.0
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     import: recipe1m.optimizers.factory
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     lr: 1e-05
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     lr_img: 1e-06
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     name: trijoint_fixed_fine_tune
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:     switch_epoch: 20
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:   path_opts: /linkhome/rech/genisi01/ugz83ue/tfood/recipe1m/options/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_noimgproj_keepall.yaml
[S 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.75:   view: ['logs:train_epoch.loss', 'logs:train_epoch.bad_pairs', 'logs:eval_epoch.metric.med_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_1_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_5_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_10_im2recipe_mean', 'logs:eval_epoch.metric.med_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_1_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_5_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_10_recipe2im_mean', 'logs:optimizer.is_optimizer_recipe&image', 'logs:optimizer.total_norm']
[I 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.76: posix.uname_result(sysname='Linux', nodename='jean-zay-ia817', release='4.18.0-305.57.1.el8_4.x86_64', version='#1 SMP Tue Jul 5 17:55:55 EDT 2022', machine='x86_64')
[I 2022-10-11 21:51:59] ...ytorch/bootstrap/run.py.80: Available GPUs: [0]
[I 2022-10-11 21:51:59] ...strap/engines/factory.py.9: Creating engine...
[I 2022-10-11 21:51:59] ...trap/datasets/factory.py.8: Creating dataset...
[I 2022-10-11 21:52:06] ...m/datasets/recipe1m.py.152: Load VCs...
[I 2022-10-11 21:52:07] ...m/datasets/recipe1m.py.165: randkw_p... None
[I 2022-10-11 21:53:08] ...rap/datasets/factory.py.21: Evaluation will take place on testset (51303 items)
[I 2022-10-11 21:53:08] ...strap/models/factory.py.21: Creating model...
[I 2022-10-11 21:53:11] ...e_networks/networks.py.203: Loading CLIP Visual model...
[I 2022-10-11 21:53:14] ...e_networks/networks.py.145: cuda
[I 2022-10-11 21:53:15] ...networks/tfood_vicha.py.49: Load VCs encoder...
[I 2022-10-11 21:53:17] ...els/metrics/trijoint.py.46: self.save_ids False
[I 2022-10-11 21:53:17] ...strap/models/factory.py.44: Enabling CUDA mode...
[I 2022-10-11 21:53:19] ...etworks/tfood_vicha.py.184: training aux kw modules
[I 2022-10-11 21:53:19] ...etworks/tfood_vicha.py.184: training aux kw modules
[I 2022-10-11 21:53:19] ...otstrap/views/factory.py.8: Creating views...
[I 2022-10-11 21:53:21] ...strap/engines/engine.py.98: Loading best_eval_epoch.metric.recall_at_1_im2recipe_mean checkpoint
[I 2022-10-11 21:53:21] ...trap/engines/engine.py.440: Loading model...
[I 2022-10-11 21:53:28] ...trap/engines/engine.py.453: Loading engine...
[I 2022-10-11 21:53:28] ...trap/engines/engine.py.108: Launching evaluation procedures
[I 2022-10-11 21:53:28] ...trap/engines/engine.py.268: Evaluating model on testset for epoch 118
[I 2022-10-11 21:53:28] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler.
[I 2022-10-11 21:53:33] ...trap/engines/engine.py.321: eval: epoch 118 | batch 0/512
[I 2022-10-11 21:53:33] ...trap/engines/engine.py.325:       elapsed: 0:00:04 | left: 0:38:43
[I 2022-10-11 21:53:33] ...trap/engines/engine.py.326:       process: 4.53816 | load: 2.35393
[I 2022-10-11 21:53:46] ...trap/engines/engine.py.321: eval: epoch 118 | batch 10/512
[I 2022-10-11 21:53:46] ...trap/engines/engine.py.325:       elapsed: 0:00:17 | left: 0:13:41
[I 2022-10-11 21:53:46] ...trap/engines/engine.py.326:       process: 1.27898 | load: 0.00016
[I 2022-10-11 21:53:58] ...trap/engines/engine.py.321: eval: epoch 118 | batch 20/512
[I 2022-10-11 21:53:58] ...trap/engines/engine.py.325:       elapsed: 0:00:30 | left: 0:10:43
[I 2022-10-11 21:53:58] ...trap/engines/engine.py.326:       process: 1.26369 | load: 0.00017
[I 2022-10-11 21:54:11] ...trap/engines/engine.py.321: eval: epoch 118 | batch 30/512
[I 2022-10-11 21:54:11] ...trap/engines/engine.py.325:       elapsed: 0:00:42 | left: 0:10:10
[I 2022-10-11 21:54:11] ...trap/engines/engine.py.326:       process: 1.23039 | load: 0.00016
[I 2022-10-11 21:54:24] ...trap/engines/engine.py.321: eval: epoch 118 | batch 40/512
[I 2022-10-11 21:54:24] ...trap/engines/engine.py.325:       elapsed: 0:00:55 | left: 0:09:45
[I 2022-10-11 21:54:24] ...trap/engines/engine.py.326:       process: 1.22520 | load: 0.00017
[I 2022-10-11 21:54:36] ...trap/engines/engine.py.321: eval: epoch 118 | batch 50/512
[I 2022-10-11 21:54:36] ...trap/engines/engine.py.325:       elapsed: 0:01:07 | left: 0:09:25
[I 2022-10-11 21:54:36] ...trap/engines/engine.py.326:       process: 1.23153 | load: 0.00017
[I 2022-10-11 21:54:48] ...trap/engines/engine.py.321: eval: epoch 118 | batch 60/512
[I 2022-10-11 21:54:48] ...trap/engines/engine.py.325:       elapsed: 0:01:19 | left: 0:09:13
[I 2022-10-11 21:54:48] ...trap/engines/engine.py.326:       process: 1.21094 | load: 0.00017
[I 2022-10-11 21:55:01] ...trap/engines/engine.py.321: eval: epoch 118 | batch 70/512
[I 2022-10-11 21:55:01] ...trap/engines/engine.py.325:       elapsed: 0:01:32 | left: 0:09:03
[I 2022-10-11 21:55:01] ...trap/engines/engine.py.326:       process: 1.20788 | load: 0.00017
[I 2022-10-11 21:55:13] ...trap/engines/engine.py.321: eval: epoch 118 | batch 80/512
[I 2022-10-11 21:55:13] ...trap/engines/engine.py.325:       elapsed: 0:01:44 | left: 0:08:54
[I 2022-10-11 21:55:13] ...trap/engines/engine.py.326:       process: 1.22641 | load: 0.00017
[I 2022-10-11 21:55:25] ...trap/engines/engine.py.321: eval: epoch 118 | batch 90/512
[I 2022-10-11 21:55:25] ...trap/engines/engine.py.325:       elapsed: 0:01:56 | left: 0:08:37
[I 2022-10-11 21:55:25] ...trap/engines/engine.py.326:       process: 1.22210 | load: 0.00020
[I 2022-10-11 21:55:37] ...trap/engines/engine.py.321: eval: epoch 118 | batch 100/512
[I 2022-10-11 21:55:37] ...trap/engines/engine.py.325:       elapsed: 0:02:08 | left: 0:08:14
[I 2022-10-11 21:55:37] ...trap/engines/engine.py.326:       process: 1.12044 | load: 0.00017
[I 2022-10-11 21:55:48] ...trap/engines/engine.py.321: eval: epoch 118 | batch 110/512
[I 2022-10-11 21:55:48] ...trap/engines/engine.py.325:       elapsed: 0:02:19 | left: 0:07:28
[I 2022-10-11 21:55:48] ...trap/engines/engine.py.326:       process: 1.11406 | load: 0.00017
[I 2022-10-11 21:55:59] ...trap/engines/engine.py.321: eval: epoch 118 | batch 120/512
[I 2022-10-11 21:55:59] ...trap/engines/engine.py.325:       elapsed: 0:02:30 | left: 0:07:09
[I 2022-10-11 21:55:59] ...trap/engines/engine.py.326:       process: 1.07455 | load: 0.00018
[I 2022-10-11 21:56:10] ...trap/engines/engine.py.321: eval: epoch 118 | batch 130/512
[I 2022-10-11 21:56:10] ...trap/engines/engine.py.325:       elapsed: 0:02:41 | left: 0:06:59
[I 2022-10-11 21:56:10] ...trap/engines/engine.py.326:       process: 1.08388 | load: 0.00017
[I 2022-10-11 21:56:21] ...trap/engines/engine.py.321: eval: epoch 118 | batch 140/512
[I 2022-10-11 21:56:21] ...trap/engines/engine.py.325:       elapsed: 0:02:53 | left: 0:06:49
[I 2022-10-11 21:56:21] ...trap/engines/engine.py.326:       process: 1.10876 | load: 0.00017
[I 2022-10-11 21:56:32] ...trap/engines/engine.py.321: eval: epoch 118 | batch 150/512
[I 2022-10-11 21:56:32] ...trap/engines/engine.py.325:       elapsed: 0:03:03 | left: 0:06:37
[I 2022-10-11 21:56:32] ...trap/engines/engine.py.326:       process: 1.08476 | load: 0.00018
[I 2022-10-11 21:56:43] ...trap/engines/engine.py.321: eval: epoch 118 | batch 160/512
[I 2022-10-11 21:56:43] ...trap/engines/engine.py.325:       elapsed: 0:03:14 | left: 0:06:26
[I 2022-10-11 21:56:43] ...trap/engines/engine.py.326:       process: 1.10985 | load: 0.00017
[I 2022-10-11 21:56:54] ...trap/engines/engine.py.321: eval: epoch 118 | batch 170/512
[I 2022-10-11 21:56:54] ...trap/engines/engine.py.325:       elapsed: 0:03:26 | left: 0:06:19
[I 2022-10-11 21:56:54] ...trap/engines/engine.py.326:       process: 1.11522 | load: 0.00018
[I 2022-10-11 21:57:05] ...trap/engines/engine.py.321: eval: epoch 118 | batch 180/512
[I 2022-10-11 21:57:05] ...trap/engines/engine.py.325:       elapsed: 0:03:37 | left: 0:06:07
[I 2022-10-11 21:57:05] ...trap/engines/engine.py.326:       process: 1.10206 | load: 0.00017
[I 2022-10-11 21:57:16] ...trap/engines/engine.py.321: eval: epoch 118 | batch 190/512
[I 2022-10-11 21:57:16] ...trap/engines/engine.py.325:       elapsed: 0:03:48 | left: 0:05:54
[I 2022-10-11 21:57:16] ...trap/engines/engine.py.326:       process: 1.07726 | load: 0.00018
[I 2022-10-11 21:57:27] ...trap/engines/engine.py.321: eval: epoch 118 | batch 200/512
[I 2022-10-11 21:57:27] ...trap/engines/engine.py.325:       elapsed: 0:03:59 | left: 0:05:43
[I 2022-10-11 21:57:27] ...trap/engines/engine.py.326:       process: 1.09263 | load: 0.00018
[I 2022-10-11 21:57:38] ...trap/engines/engine.py.321: eval: epoch 118 | batch 210/512
[I 2022-10-11 21:57:38] ...trap/engines/engine.py.325:       elapsed: 0:04:10 | left: 0:05:32
[I 2022-10-11 21:57:38] ...trap/engines/engine.py.326:       process: 1.10033 | load: 0.00017
[I 2022-10-11 21:57:50] ...trap/engines/engine.py.321: eval: epoch 118 | batch 220/512
[I 2022-10-11 21:57:50] ...trap/engines/engine.py.325:       elapsed: 0:04:21 | left: 0:05:21
[I 2022-10-11 21:57:50] ...trap/engines/engine.py.326:       process: 1.11296 | load: 0.00019
[I 2022-10-11 21:58:01] ...trap/engines/engine.py.321: eval: epoch 118 | batch 230/512
[I 2022-10-11 21:58:01] ...trap/engines/engine.py.325:       elapsed: 0:04:32 | left: 0:05:10
[I 2022-10-11 21:58:01] ...trap/engines/engine.py.326:       process: 1.09714 | load: 0.00018
[I 2022-10-11 21:58:12] ...trap/engines/engine.py.321: eval: epoch 118 | batch 240/512
[I 2022-10-11 21:58:12] ...trap/engines/engine.py.325:       elapsed: 0:04:43 | left: 0:04:59
[I 2022-10-11 21:58:12] ...trap/engines/engine.py.326:       process: 1.09706 | load: 0.00018
[I 2022-10-11 21:58:23] ...trap/engines/engine.py.321: eval: epoch 118 | batch 250/512
[I 2022-10-11 21:58:23] ...trap/engines/engine.py.325:       elapsed: 0:04:54 | left: 0:04:48
[I 2022-10-11 21:58:23] ...trap/engines/engine.py.326:       process: 1.09862 | load: 0.00018
[I 2022-10-11 21:58:34] ...trap/engines/engine.py.321: eval: epoch 118 | batch 260/512
[I 2022-10-11 21:58:34] ...trap/engines/engine.py.325:       elapsed: 0:05:05 | left: 0:04:35
[I 2022-10-11 21:58:34] ...trap/engines/engine.py.326:       process: 1.09471 | load: 0.00018
[I 2022-10-11 21:58:44] ...trap/engines/engine.py.321: eval: epoch 118 | batch 270/512
[I 2022-10-11 21:58:44] ...trap/engines/engine.py.325:       elapsed: 0:05:16 | left: 0:04:23
[I 2022-10-11 21:58:44] ...trap/engines/engine.py.326:       process: 1.07004 | load: 0.00017
[I 2022-10-11 21:58:55] ...trap/engines/engine.py.321: eval: epoch 118 | batch 280/512
[I 2022-10-11 21:58:55] ...trap/engines/engine.py.325:       elapsed: 0:05:27 | left: 0:04:15
[I 2022-10-11 21:58:55] ...trap/engines/engine.py.326:       process: 1.11799 | load: 0.00020
[I 2022-10-11 21:59:06] ...trap/engines/engine.py.321: eval: epoch 118 | batch 290/512
[I 2022-10-11 21:59:06] ...trap/engines/engine.py.325:       elapsed: 0:05:38 | left: 0:04:04
[I 2022-10-11 21:59:06] ...trap/engines/engine.py.326:       process: 1.10631 | load: 0.00019
[I 2022-10-11 21:59:17] ...trap/engines/engine.py.321: eval: epoch 118 | batch 300/512
[I 2022-10-11 21:59:17] ...trap/engines/engine.py.325:       elapsed: 0:05:49 | left: 0:03:53
[I 2022-10-11 21:59:17] ...trap/engines/engine.py.326:       process: 1.11747 | load: 0.00018
[I 2022-10-11 21:59:29] ...trap/engines/engine.py.321: eval: epoch 118 | batch 310/512
[I 2022-10-11 21:59:29] ...trap/engines/engine.py.325:       elapsed: 0:06:00 | left: 0:03:43
[I 2022-10-11 21:59:29] ...trap/engines/engine.py.326:       process: 1.11468 | load: 0.00018
[I 2022-10-11 21:59:40] ...trap/engines/engine.py.321: eval: epoch 118 | batch 320/512
[I 2022-10-11 21:59:40] ...trap/engines/engine.py.325:       elapsed: 0:06:11 | left: 0:03:30
[I 2022-10-11 21:59:40] ...trap/engines/engine.py.326:       process: 1.08733 | load: 0.00019
[I 2022-10-11 21:59:51] ...trap/engines/engine.py.321: eval: epoch 118 | batch 330/512
[I 2022-10-11 21:59:51] ...trap/engines/engine.py.325:       elapsed: 0:06:22 | left: 0:03:22
[I 2022-10-11 21:59:51] ...trap/engines/engine.py.326:       process: 1.11815 | load: 0.00018
[I 2022-10-11 22:00:02] ...trap/engines/engine.py.321: eval: epoch 118 | batch 340/512
[I 2022-10-11 22:00:02] ...trap/engines/engine.py.325:       elapsed: 0:06:33 | left: 0:03:10
[I 2022-10-11 22:00:02] ...trap/engines/engine.py.326:       process: 1.10198 | load: 0.00018
[I 2022-10-11 22:00:13] ...trap/engines/engine.py.321: eval: epoch 118 | batch 350/512
[I 2022-10-11 22:00:13] ...trap/engines/engine.py.325:       elapsed: 0:06:44 | left: 0:02:59
[I 2022-10-11 22:00:13] ...trap/engines/engine.py.326:       process: 1.09810 | load: 0.00019
[I 2022-10-11 22:00:24] ...trap/engines/engine.py.321: eval: epoch 118 | batch 360/512
[I 2022-10-11 22:00:24] ...trap/engines/engine.py.325:       elapsed: 0:06:55 | left: 0:02:48
[I 2022-10-11 22:00:24] ...trap/engines/engine.py.326:       process: 1.11774 | load: 0.00020
[I 2022-10-11 22:00:35] ...trap/engines/engine.py.321: eval: epoch 118 | batch 370/512
[I 2022-10-11 22:00:35] ...trap/engines/engine.py.325:       elapsed: 0:07:06 | left: 0:02:37
[I 2022-10-11 22:00:35] ...trap/engines/engine.py.326:       process: 1.09239 | load: 0.00019
[I 2022-10-11 22:00:46] ...trap/engines/engine.py.321: eval: epoch 118 | batch 380/512
[I 2022-10-11 22:00:46] ...trap/engines/engine.py.325:       elapsed: 0:07:17 | left: 0:02:26
[I 2022-10-11 22:00:46] ...trap/engines/engine.py.326:       process: 1.10802 | load: 0.00019
[I 2022-10-11 22:00:57] ...trap/engines/engine.py.321: eval: epoch 118 | batch 390/512
[I 2022-10-11 22:00:57] ...trap/engines/engine.py.325:       elapsed: 0:07:28 | left: 0:02:15
[I 2022-10-11 22:00:57] ...trap/engines/engine.py.326:       process: 1.11420 | load: 0.00019
[I 2022-10-11 22:01:08] ...trap/engines/engine.py.321: eval: epoch 118 | batch 400/512
[I 2022-10-11 22:01:08] ...trap/engines/engine.py.325:       elapsed: 0:07:39 | left: 0:02:02
[I 2022-10-11 22:01:08] ...trap/engines/engine.py.326:       process: 1.08097 | load: 0.00019
[I 2022-10-11 22:01:19] ...trap/engines/engine.py.321: eval: epoch 118 | batch 410/512
[I 2022-10-11 22:01:19] ...trap/engines/engine.py.325:       elapsed: 0:07:50 | left: 0:01:52
[I 2022-10-11 22:01:19] ...trap/engines/engine.py.326:       process: 1.11015 | load: 0.00019
[I 2022-10-11 22:01:30] ...trap/engines/engine.py.321: eval: epoch 118 | batch 420/512
[I 2022-10-11 22:01:30] ...trap/engines/engine.py.325:       elapsed: 0:08:01 | left: 0:01:41
[I 2022-10-11 22:01:30] ...trap/engines/engine.py.326:       process: 1.12501 | load: 0.00020
[I 2022-10-11 22:01:41] ...trap/engines/engine.py.321: eval: epoch 118 | batch 430/512
[I 2022-10-11 22:01:41] ...trap/engines/engine.py.325:       elapsed: 0:08:12 | left: 0:01:30
[I 2022-10-11 22:01:41] ...trap/engines/engine.py.326:       process: 1.11556 | load: 0.00019
[I 2022-10-11 22:01:52] ...trap/engines/engine.py.321: eval: epoch 118 | batch 440/512
[I 2022-10-11 22:01:52] ...trap/engines/engine.py.325:       elapsed: 0:08:23 | left: 0:01:19
[I 2022-10-11 22:01:52] ...trap/engines/engine.py.326:       process: 1.11614 | load: 0.00019
[I 2022-10-11 22:02:03] ...trap/engines/engine.py.321: eval: epoch 118 | batch 450/512
[I 2022-10-11 22:02:03] ...trap/engines/engine.py.325:       elapsed: 0:08:35 | left: 0:01:08
[I 2022-10-11 22:02:03] ...trap/engines/engine.py.326:       process: 1.11593 | load: 0.00019
[I 2022-10-11 22:02:14] ...trap/engines/engine.py.321: eval: epoch 118 | batch 460/512
[I 2022-10-11 22:02:14] ...trap/engines/engine.py.325:       elapsed: 0:08:46 | left: 0:00:57
[I 2022-10-11 22:02:14] ...trap/engines/engine.py.326:       process: 1.09500 | load: 0.00018
[I 2022-10-11 22:02:25] ...trap/engines/engine.py.321: eval: epoch 118 | batch 470/512
[I 2022-10-11 22:02:25] ...trap/engines/engine.py.325:       elapsed: 0:08:57 | left: 0:00:45
[I 2022-10-11 22:02:25] ...trap/engines/engine.py.326:       process: 1.08739 | load: 0.00019
[I 2022-10-11 22:02:37] ...trap/engines/engine.py.321: eval: epoch 118 | batch 480/512
[I 2022-10-11 22:02:37] ...trap/engines/engine.py.325:       elapsed: 0:09:08 | left: 0:00:35
[I 2022-10-11 22:02:37] ...trap/engines/engine.py.326:       process: 1.09733 | load: 0.00019
[I 2022-10-11 22:02:48] ...trap/engines/engine.py.321: eval: epoch 118 | batch 490/512
[I 2022-10-11 22:02:48] ...trap/engines/engine.py.325:       elapsed: 0:09:19 | left: 0:00:24
[I 2022-10-11 22:02:48] ...trap/engines/engine.py.326:       process: 1.11146 | load: 0.00019
[I 2022-10-11 22:02:59] ...trap/engines/engine.py.321: eval: epoch 118 | batch 500/512
[I 2022-10-11 22:02:59] ...trap/engines/engine.py.325:       elapsed: 0:09:30 | left: 0:00:13
[I 2022-10-11 22:02:59] ...trap/engines/engine.py.326:       process: 1.11364 | load: 0.00019
[I 2022-10-11 22:03:10] ...trap/engines/engine.py.321: eval: epoch 118 | batch 510/512
[I 2022-10-11 22:03:10] ...trap/engines/engine.py.325:       elapsed: 0:09:41 | left: 0:00:02
[I 2022-10-11 22:03:10] ...trap/engines/engine.py.326:       process: 1.10799 | load: 0.00015
[S 2022-10-11 22:03:12] ...trap/engines/engine.py.113: eval_epoch.epoch: 118
[I 2022-10-11 22:03:12] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 10 x 1000 matchs
[S 2022-10-11 22:03:38] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 1.00
[S 2022-10-11 22:03:38] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 1.00
[S 2022-10-11 22:03:38] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.756000
[S 2022-10-11 22:03:38] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.911800
[S 2022-10-11 22:03:38] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.936400
[S 2022-10-11 22:03:38] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.748800
[S 2022-10-11 22:03:38] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.914100
[S 2022-10-11 22:03:38] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.936900
[S 2022-10-11 22:03:38] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000
[S 2022-10-11 22:03:38] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000
[S 2022-10-11 22:03:38] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.012681
[S 2022-10-11 22:03:38] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.007181
[S 2022-10-11 22:03:38] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.007723
[S 2022-10-11 22:03:38] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.011931
[S 2022-10-11 22:03:38] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.007739
[S 2022-10-11 22:03:38] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.008654
[I 2022-10-11 22:03:53] ...trap/engines/engine.py.115: Ending evaluation procedures
[W 2022-10-11 22:03:55] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json'
[W 2022-10-11 22:03:55] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[W 2022-10-11 22:03:55] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[I 2022-10-11 22:04:03] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html
[I 2022-10-11 22:06:08] ...ytorch/bootstrap/run.py.73: Saving environment info
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75: options: 
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:   dataset:
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     aux_kw_path:
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       test: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_test_titles_kw.json
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       train: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_train_titles_kw.json
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       val: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_val_titles_kw.json
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     batch_sampler: triplet_classif
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     batch_size: 100
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     database: lmdb
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     dir: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     eval_split: test
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     freq_mismatch: 0.0
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     image_from: database
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     import: recipe1m.datasets.factory
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     kw_path:
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       test: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_test_ingr_kw.json
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       train: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_train_ingr_kw.json
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       val: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_val_ingr_kw.json
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     name: recipe1m
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     nb_classes: 1048
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     nb_threads: 4
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     randkw_p: 0.3
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     randkw_p_aux: 0.5
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     tokenized_raw_text: True
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     train_split: None
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     use_vcs: True
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:   engine:
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     name: logger
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     nb_epochs: 120
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     print_freq: 10
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     saving_criteria: ['eval_epoch.metric.med_im2recipe_mean:min', 'eval_epoch.metric.recall_at_1_im2recipe_mean:max']
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:   exp:
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     checkpoint: None
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     dir: /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     resume: best_eval_epoch.metric.recall_at_1_im2recipe_mean
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:   misc:
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     cuda: True
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     data_parrallel: True
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     logs_name: None
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     overrite: False
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     seed: 1338
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:   model:
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     criterion:
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:         tri_cross_encoder: False
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       itm_loss_weight: 1
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       keep_background: False
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       retrieval_strategy:
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:         aggregation: valid
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:         get_tokens: True
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:         margin: 0.05
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:         margin_params:
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:           increment: 0.005
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:           increment_margin: True
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:           max_margin: 0.3
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:         name: triplet
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:         nb_samples: 9999
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:         sampling: max_negative
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:         substrategy: ['IRR', 'RII', 'SIRR', 'SRII']
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:         substrategy_weights: [1.0, 1.0, 0.1, 0.1]
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     import: recipe1m.models.factory
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     metric:
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       k_test: 10
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       nb_bags: 5
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       nb_matchs_per_bag: 10000
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       retrieval_dir: /gpfsscratch/rech/dyf/ugz83ue/tmp/recipe1m
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       trijoint: True
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     name: trijoint
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     network:
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       activations: ['tanh', 'normalize']
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       aux_kwords: True
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       aux_kwords_encoder: True
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       avg_concat: True
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       bert_config: /linkhome/rech/genisi01/ugz83ue/tfood/recipe1m/models/networks/recipe_networks/config_bert_albef.json
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       cat_pos: 0
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       checkpoint: None
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       class_attention: False
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       class_attention_fus: False
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       cls_token: False
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       cls_token_fus: False
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       cross_decoder_image: True
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       cross_decoder_recipe: True
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:         class_attention: False
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:         cls_token: False
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:         cross_attention: False
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:         cross_decoder: True
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:         double_decoder: False
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:         get_tokens: False
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:         load_backbones: False
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:         n_heads: 4
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:         n_layers: 4
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:         query_img: False
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       cross_transformer: False
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       dim_emb: 1024
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       dim_image_out: 2048
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       dim_ingr_out: 300
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       dim_instr_in: 1024
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       dim_instr_out: 1024
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       embedding_names: ['proj_image']
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       freeze_im: False
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       freeze_rec: False
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       get_tokens: True
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       get_tokens_cross_decoder_recipe: True
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       hidden_size: 512
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       image_backbone_name: clip
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       image_res: 224
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       kwords_same_level: False
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       last_cat_aux: True
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       n_heads: 4
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       n_heads_cross: 4
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       n_heads_cross_image: 4
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       n_heads_single: 4
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       n_layers: 2
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       n_layers_cross: 2
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       n_layers_cross_image: 1
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       n_layers_single: 2
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       num_hidden_layers_kw: 2
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       old: True
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       path_ingrs: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/text/vocab.pkl
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       path_vocab: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/text/vocab_all.txt
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       recipe_encoder: tfoodvicha-transformer
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       text_encoder: bert-base-uncased
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       vision_width: 768
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       vit_all_tokens: True
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       with_ingrs: True
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       with_instrs: True
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:       with_titles: True
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     with_classif: False
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:   optimizer:
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     batch_size_factor: None
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     clip_grad: 8.0
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     import: recipe1m.optimizers.factory
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     lr: 1e-05
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     lr_img: 1e-06
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     name: trijoint_fixed_fine_tune
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:     switch_epoch: 20
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:   path_opts: /linkhome/rech/genisi01/ugz83ue/tfood/recipe1m/options/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_noimgproj_keepall.yaml
[S 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.75:   view: ['logs:train_epoch.loss', 'logs:train_epoch.bad_pairs', 'logs:eval_epoch.metric.med_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_1_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_5_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_10_im2recipe_mean', 'logs:eval_epoch.metric.med_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_1_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_5_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_10_recipe2im_mean', 'logs:optimizer.is_optimizer_recipe&image', 'logs:optimizer.total_norm']
[I 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.76: posix.uname_result(sysname='Linux', nodename='jean-zay-ia817', release='4.18.0-305.57.1.el8_4.x86_64', version='#1 SMP Tue Jul 5 17:55:55 EDT 2022', machine='x86_64')
[I 2022-10-11 22:06:11] ...ytorch/bootstrap/run.py.80: Available GPUs: [0]
[I 2022-10-11 22:06:11] ...strap/engines/factory.py.9: Creating engine...
[I 2022-10-11 22:06:11] ...trap/datasets/factory.py.8: Creating dataset...
[I 2022-10-11 22:06:18] ...m/datasets/recipe1m.py.152: Load VCs...
[I 2022-10-11 22:06:19] ...m/datasets/recipe1m.py.165: randkw_p... None
[I 2022-10-11 22:07:21] ...rap/datasets/factory.py.21: Evaluation will take place on testset (51303 items)
[I 2022-10-11 22:07:21] ...strap/models/factory.py.21: Creating model...
[I 2022-10-11 22:07:24] ...e_networks/networks.py.203: Loading CLIP Visual model...
[I 2022-10-11 22:07:27] ...e_networks/networks.py.145: cuda
[I 2022-10-11 22:07:28] ...networks/tfood_vicha.py.49: Load VCs encoder...
[I 2022-10-11 22:07:30] ...els/metrics/trijoint.py.46: self.save_ids False
[I 2022-10-11 22:07:30] ...strap/models/factory.py.44: Enabling CUDA mode...
[I 2022-10-11 22:07:32] ...etworks/tfood_vicha.py.184: training aux kw modules
[I 2022-10-11 22:07:32] ...etworks/tfood_vicha.py.184: training aux kw modules
[I 2022-10-11 22:07:32] ...otstrap/views/factory.py.8: Creating views...
[I 2022-10-11 22:07:34] ...strap/engines/engine.py.98: Loading best_eval_epoch.metric.recall_at_1_im2recipe_mean checkpoint
[I 2022-10-11 22:07:34] ...trap/engines/engine.py.440: Loading model...
[I 2022-10-11 22:07:41] ...trap/engines/engine.py.453: Loading engine...
[I 2022-10-11 22:07:41] ...trap/engines/engine.py.108: Launching evaluation procedures
[I 2022-10-11 22:07:41] ...trap/engines/engine.py.268: Evaluating model on testset for epoch 118
[I 2022-10-11 22:07:41] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler.
[I 2022-10-11 22:07:45] ...trap/engines/engine.py.321: eval: epoch 118 | batch 0/512
[I 2022-10-11 22:07:45] ...trap/engines/engine.py.325:       elapsed: 0:00:04 | left: 0:35:39
[I 2022-10-11 22:07:45] ...trap/engines/engine.py.326:       process: 4.17793 | load: 2.29789
[I 2022-10-11 22:07:57] ...trap/engines/engine.py.321: eval: epoch 118 | batch 10/512
[I 2022-10-11 22:07:57] ...trap/engines/engine.py.325:       elapsed: 0:00:16 | left: 0:12:34
[I 2022-10-11 22:07:57] ...trap/engines/engine.py.326:       process: 1.19672 | load: 0.00016
[I 2022-10-11 22:08:09] ...trap/engines/engine.py.321: eval: epoch 118 | batch 20/512
[I 2022-10-11 22:08:09] ...trap/engines/engine.py.325:       elapsed: 0:00:27 | left: 0:09:50
[I 2022-10-11 22:08:09] ...trap/engines/engine.py.326:       process: 1.14987 | load: 0.00016
[I 2022-10-11 22:08:20] ...trap/engines/engine.py.321: eval: epoch 118 | batch 30/512
[I 2022-10-11 22:08:20] ...trap/engines/engine.py.325:       elapsed: 0:00:39 | left: 0:09:27
[I 2022-10-11 22:08:20] ...trap/engines/engine.py.326:       process: 1.19814 | load: 0.00017
[I 2022-10-11 22:08:32] ...trap/engines/engine.py.321: eval: epoch 118 | batch 40/512
[I 2022-10-11 22:08:32] ...trap/engines/engine.py.325:       elapsed: 0:00:51 | left: 0:09:17
[I 2022-10-11 22:08:32] ...trap/engines/engine.py.326:       process: 1.16996 | load: 0.00017
[I 2022-10-11 22:08:44] ...trap/engines/engine.py.321: eval: epoch 118 | batch 50/512
[I 2022-10-11 22:08:44] ...trap/engines/engine.py.325:       elapsed: 0:01:03 | left: 0:09:07
[I 2022-10-11 22:08:44] ...trap/engines/engine.py.326:       process: 1.20691 | load: 0.00020
[I 2022-10-11 22:08:56] ...trap/engines/engine.py.321: eval: epoch 118 | batch 60/512
[I 2022-10-11 22:08:56] ...trap/engines/engine.py.325:       elapsed: 0:01:14 | left: 0:08:55
[I 2022-10-11 22:08:56] ...trap/engines/engine.py.326:       process: 1.17747 | load: 0.00017
[I 2022-10-11 22:09:08] ...trap/engines/engine.py.321: eval: epoch 118 | batch 70/512
[I 2022-10-11 22:09:08] ...trap/engines/engine.py.325:       elapsed: 0:01:26 | left: 0:08:46
[I 2022-10-11 22:09:08] ...trap/engines/engine.py.326:       process: 1.19036 | load: 0.00018
[I 2022-10-11 22:09:20] ...trap/engines/engine.py.321: eval: epoch 118 | batch 80/512
[I 2022-10-11 22:09:20] ...trap/engines/engine.py.325:       elapsed: 0:01:38 | left: 0:08:38
[I 2022-10-11 22:09:20] ...trap/engines/engine.py.326:       process: 1.19703 | load: 0.00017
[I 2022-10-11 22:09:32] ...trap/engines/engine.py.321: eval: epoch 118 | batch 90/512
[I 2022-10-11 22:09:32] ...trap/engines/engine.py.325:       elapsed: 0:01:50 | left: 0:08:21
[I 2022-10-11 22:09:32] ...trap/engines/engine.py.326:       process: 1.18695 | load: 0.00019
[I 2022-10-11 22:09:44] ...trap/engines/engine.py.321: eval: epoch 118 | batch 100/512
[I 2022-10-11 22:09:44] ...trap/engines/engine.py.325:       elapsed: 0:02:02 | left: 0:08:13
[I 2022-10-11 22:09:44] ...trap/engines/engine.py.326:       process: 1.23170 | load: 0.00019
[I 2022-10-11 22:09:56] ...trap/engines/engine.py.321: eval: epoch 118 | batch 110/512
[I 2022-10-11 22:09:56] ...trap/engines/engine.py.325:       elapsed: 0:02:15 | left: 0:08:18
[I 2022-10-11 22:09:56] ...trap/engines/engine.py.326:       process: 1.24456 | load: 0.00017
[I 2022-10-11 22:10:09] ...trap/engines/engine.py.321: eval: epoch 118 | batch 120/512
[I 2022-10-11 22:10:09] ...trap/engines/engine.py.325:       elapsed: 0:02:27 | left: 0:08:07
[I 2022-10-11 22:10:09] ...trap/engines/engine.py.326:       process: 1.23624 | load: 0.00017
[I 2022-10-11 22:10:21] ...trap/engines/engine.py.321: eval: epoch 118 | batch 130/512
[I 2022-10-11 22:10:21] ...trap/engines/engine.py.325:       elapsed: 0:02:40 | left: 0:07:54
[I 2022-10-11 22:10:21] ...trap/engines/engine.py.326:       process: 1.21207 | load: 0.00017
[I 2022-10-11 22:10:34] ...trap/engines/engine.py.321: eval: epoch 118 | batch 140/512
[I 2022-10-11 22:10:34] ...trap/engines/engine.py.325:       elapsed: 0:02:52 | left: 0:07:47
[I 2022-10-11 22:10:34] ...trap/engines/engine.py.326:       process: 1.26842 | load: 0.00017
[I 2022-10-11 22:10:46] ...trap/engines/engine.py.321: eval: epoch 118 | batch 150/512
[I 2022-10-11 22:10:46] ...trap/engines/engine.py.325:       elapsed: 0:03:05 | left: 0:07:29
[I 2022-10-11 22:10:46] ...trap/engines/engine.py.326:       process: 1.22118 | load: 0.00017
[I 2022-10-11 22:10:58] ...trap/engines/engine.py.321: eval: epoch 118 | batch 160/512
[I 2022-10-11 22:10:58] ...trap/engines/engine.py.325:       elapsed: 0:03:17 | left: 0:07:14
[I 2022-10-11 22:10:58] ...trap/engines/engine.py.326:       process: 1.24095 | load: 0.00019
[I 2022-10-11 22:11:11] ...trap/engines/engine.py.321: eval: epoch 118 | batch 170/512
[I 2022-10-11 22:11:11] ...trap/engines/engine.py.325:       elapsed: 0:03:29 | left: 0:07:06
[I 2022-10-11 22:11:11] ...trap/engines/engine.py.326:       process: 1.25269 | load: 0.00018
[I 2022-10-11 22:11:24] ...trap/engines/engine.py.321: eval: epoch 118 | batch 180/512
[I 2022-10-11 22:11:24] ...trap/engines/engine.py.325:       elapsed: 0:03:42 | left: 0:06:59
[I 2022-10-11 22:11:24] ...trap/engines/engine.py.326:       process: 1.28468 | load: 0.00021
[I 2022-10-11 22:11:36] ...trap/engines/engine.py.321: eval: epoch 118 | batch 190/512
[I 2022-10-11 22:11:36] ...trap/engines/engine.py.325:       elapsed: 0:03:55 | left: 0:06:52
[I 2022-10-11 22:11:36] ...trap/engines/engine.py.326:       process: 1.26133 | load: 0.00018
[I 2022-10-11 22:11:49] ...trap/engines/engine.py.321: eval: epoch 118 | batch 200/512
[I 2022-10-11 22:11:49] ...trap/engines/engine.py.325:       elapsed: 0:04:08 | left: 0:06:39
[I 2022-10-11 22:11:49] ...trap/engines/engine.py.326:       process: 1.26465 | load: 0.00017
[I 2022-10-11 22:12:02] ...trap/engines/engine.py.321: eval: epoch 118 | batch 210/512
[I 2022-10-11 22:12:02] ...trap/engines/engine.py.325:       elapsed: 0:04:21 | left: 0:06:27
[I 2022-10-11 22:12:02] ...trap/engines/engine.py.326:       process: 1.25736 | load: 0.00019
[I 2022-10-11 22:12:15] ...trap/engines/engine.py.321: eval: epoch 118 | batch 220/512
[I 2022-10-11 22:12:15] ...trap/engines/engine.py.325:       elapsed: 0:04:33 | left: 0:06:11
[I 2022-10-11 22:12:15] ...trap/engines/engine.py.326:       process: 1.27312 | load: 0.00016
[I 2022-10-11 22:12:28] ...trap/engines/engine.py.321: eval: epoch 118 | batch 230/512
[I 2022-10-11 22:12:28] ...trap/engines/engine.py.325:       elapsed: 0:04:46 | left: 0:05:58
[I 2022-10-11 22:12:28] ...trap/engines/engine.py.326:       process: 1.27291 | load: 0.00017
[I 2022-10-11 22:12:40] ...trap/engines/engine.py.321: eval: epoch 118 | batch 240/512
[I 2022-10-11 22:12:40] ...trap/engines/engine.py.325:       elapsed: 0:04:59 | left: 0:05:47
[I 2022-10-11 22:12:40] ...trap/engines/engine.py.326:       process: 1.26649 | load: 0.00017
[I 2022-10-11 22:12:53] ...trap/engines/engine.py.321: eval: epoch 118 | batch 250/512
[I 2022-10-11 22:12:53] ...trap/engines/engine.py.325:       elapsed: 0:05:12 | left: 0:05:37
[I 2022-10-11 22:12:53] ...trap/engines/engine.py.326:       process: 1.28328 | load: 0.00018
[I 2022-10-11 22:13:06] ...trap/engines/engine.py.321: eval: epoch 118 | batch 260/512
[I 2022-10-11 22:13:06] ...trap/engines/engine.py.325:       elapsed: 0:05:25 | left: 0:05:21
[I 2022-10-11 22:13:06] ...trap/engines/engine.py.326:       process: 1.26206 | load: 0.00020
[I 2022-10-11 22:13:19] ...trap/engines/engine.py.321: eval: epoch 118 | batch 270/512
[I 2022-10-11 22:13:19] ...trap/engines/engine.py.325:       elapsed: 0:05:37 | left: 0:05:05
[I 2022-10-11 22:13:19] ...trap/engines/engine.py.326:       process: 1.22072 | load: 0.00021
[I 2022-10-11 22:13:32] ...trap/engines/engine.py.321: eval: epoch 118 | batch 280/512
[I 2022-10-11 22:13:32] ...trap/engines/engine.py.325:       elapsed: 0:05:50 | left: 0:04:59
[I 2022-10-11 22:13:32] ...trap/engines/engine.py.326:       process: 1.33655 | load: 0.00018
[I 2022-10-11 22:13:44] ...trap/engines/engine.py.321: eval: epoch 118 | batch 290/512
[I 2022-10-11 22:13:44] ...trap/engines/engine.py.325:       elapsed: 0:06:03 | left: 0:04:45
[I 2022-10-11 22:13:44] ...trap/engines/engine.py.326:       process: 1.28884 | load: 0.00018
[I 2022-10-11 22:13:57] ...trap/engines/engine.py.321: eval: epoch 118 | batch 300/512
[I 2022-10-11 22:13:57] ...trap/engines/engine.py.325:       elapsed: 0:06:16 | left: 0:04:34
[I 2022-10-11 22:13:57] ...trap/engines/engine.py.326:       process: 1.32716 | load: 0.00017
[I 2022-10-11 22:14:10] ...trap/engines/engine.py.321: eval: epoch 118 | batch 310/512
[I 2022-10-11 22:14:10] ...trap/engines/engine.py.325:       elapsed: 0:06:29 | left: 0:04:20
[I 2022-10-11 22:14:10] ...trap/engines/engine.py.326:       process: 1.28180 | load: 0.00018
[I 2022-10-11 22:14:23] ...trap/engines/engine.py.321: eval: epoch 118 | batch 320/512
[I 2022-10-11 22:14:23] ...trap/engines/engine.py.325:       elapsed: 0:06:42 | left: 0:04:07
[I 2022-10-11 22:14:23] ...trap/engines/engine.py.326:       process: 1.30173 | load: 0.00018
[I 2022-10-11 22:14:36] ...trap/engines/engine.py.321: eval: epoch 118 | batch 330/512
[I 2022-10-11 22:14:36] ...trap/engines/engine.py.325:       elapsed: 0:06:55 | left: 0:03:55
[I 2022-10-11 22:14:36] ...trap/engines/engine.py.326:       process: 1.29387 | load: 0.00017
[I 2022-10-11 22:14:49] ...trap/engines/engine.py.321: eval: epoch 118 | batch 340/512
[I 2022-10-11 22:14:49] ...trap/engines/engine.py.325:       elapsed: 0:07:08 | left: 0:03:43
[I 2022-10-11 22:14:49] ...trap/engines/engine.py.326:       process: 1.27936 | load: 0.00017
[I 2022-10-11 22:15:02] ...trap/engines/engine.py.321: eval: epoch 118 | batch 350/512
[I 2022-10-11 22:15:02] ...trap/engines/engine.py.325:       elapsed: 0:07:20 | left: 0:03:27
[I 2022-10-11 22:15:02] ...trap/engines/engine.py.326:       process: 1.25544 | load: 0.00018
[I 2022-10-11 22:15:15] ...trap/engines/engine.py.321: eval: epoch 118 | batch 360/512
[I 2022-10-11 22:15:15] ...trap/engines/engine.py.325:       elapsed: 0:07:33 | left: 0:03:16
[I 2022-10-11 22:15:15] ...trap/engines/engine.py.326:       process: 1.30295 | load: 0.00018
[I 2022-10-11 22:15:28] ...trap/engines/engine.py.321: eval: epoch 118 | batch 370/512
[I 2022-10-11 22:15:28] ...trap/engines/engine.py.325:       elapsed: 0:07:46 | left: 0:03:03
[I 2022-10-11 22:15:28] ...trap/engines/engine.py.326:       process: 1.25271 | load: 0.00018
[I 2022-10-11 22:15:41] ...trap/engines/engine.py.321: eval: epoch 118 | batch 380/512
[I 2022-10-11 22:15:41] ...trap/engines/engine.py.325:       elapsed: 0:07:59 | left: 0:02:47
[I 2022-10-11 22:15:41] ...trap/engines/engine.py.326:       process: 1.27000 | load: 0.00018
[I 2022-10-11 22:15:54] ...trap/engines/engine.py.321: eval: epoch 118 | batch 390/512
[I 2022-10-11 22:15:54] ...trap/engines/engine.py.325:       elapsed: 0:08:12 | left: 0:02:37
[I 2022-10-11 22:15:54] ...trap/engines/engine.py.326:       process: 1.27037 | load: 0.00019
[I 2022-10-11 22:16:06] ...trap/engines/engine.py.321: eval: epoch 118 | batch 400/512
[I 2022-10-11 22:16:06] ...trap/engines/engine.py.325:       elapsed: 0:08:25 | left: 0:02:21
[I 2022-10-11 22:16:06] ...trap/engines/engine.py.326:       process: 1.26017 | load: 0.00017
[I 2022-10-11 22:16:19] ...trap/engines/engine.py.321: eval: epoch 118 | batch 410/512
[I 2022-10-11 22:16:19] ...trap/engines/engine.py.325:       elapsed: 0:08:38 | left: 0:02:10
[I 2022-10-11 22:16:19] ...trap/engines/engine.py.326:       process: 1.28039 | load: 0.00018
[I 2022-10-11 22:16:32] ...trap/engines/engine.py.321: eval: epoch 118 | batch 420/512
[I 2022-10-11 22:16:32] ...trap/engines/engine.py.325:       elapsed: 0:08:51 | left: 0:02:00
[I 2022-10-11 22:16:32] ...trap/engines/engine.py.326:       process: 1.32764 | load: 0.00019
[I 2022-10-11 22:16:45] ...trap/engines/engine.py.321: eval: epoch 118 | batch 430/512
[I 2022-10-11 22:16:45] ...trap/engines/engine.py.325:       elapsed: 0:09:03 | left: 0:01:43
[I 2022-10-11 22:16:45] ...trap/engines/engine.py.326:       process: 1.23755 | load: 0.00019
[I 2022-10-11 22:16:57] ...trap/engines/engine.py.321: eval: epoch 118 | batch 440/512
[I 2022-10-11 22:16:57] ...trap/engines/engine.py.325:       elapsed: 0:09:16 | left: 0:01:30
[I 2022-10-11 22:16:57] ...trap/engines/engine.py.326:       process: 1.24677 | load: 0.00019
[I 2022-10-11 22:17:10] ...trap/engines/engine.py.321: eval: epoch 118 | batch 450/512
[I 2022-10-11 22:17:10] ...trap/engines/engine.py.325:       elapsed: 0:09:28 | left: 0:01:17
[I 2022-10-11 22:17:10] ...trap/engines/engine.py.326:       process: 1.22980 | load: 0.00018
[I 2022-10-11 22:17:22] ...trap/engines/engine.py.321: eval: epoch 118 | batch 460/512
[I 2022-10-11 22:17:22] ...trap/engines/engine.py.325:       elapsed: 0:09:41 | left: 0:01:04
[I 2022-10-11 22:17:22] ...trap/engines/engine.py.326:       process: 1.22054 | load: 0.00018
[I 2022-10-11 22:17:35] ...trap/engines/engine.py.321: eval: epoch 118 | batch 470/512
[I 2022-10-11 22:17:35] ...trap/engines/engine.py.325:       elapsed: 0:09:53 | left: 0:00:51
[I 2022-10-11 22:17:35] ...trap/engines/engine.py.326:       process: 1.21272 | load: 0.00018
[I 2022-10-11 22:17:47] ...trap/engines/engine.py.321: eval: epoch 118 | batch 480/512
[I 2022-10-11 22:17:47] ...trap/engines/engine.py.325:       elapsed: 0:10:06 | left: 0:00:39
[I 2022-10-11 22:17:47] ...trap/engines/engine.py.326:       process: 1.22246 | load: 0.00018
[I 2022-10-11 22:18:00] ...trap/engines/engine.py.321: eval: epoch 118 | batch 490/512
[I 2022-10-11 22:18:00] ...trap/engines/engine.py.325:       elapsed: 0:10:18 | left: 0:00:27
[I 2022-10-11 22:18:00] ...trap/engines/engine.py.326:       process: 1.23708 | load: 0.00018
[I 2022-10-11 22:18:12] ...trap/engines/engine.py.321: eval: epoch 118 | batch 500/512
[I 2022-10-11 22:18:12] ...trap/engines/engine.py.325:       elapsed: 0:10:31 | left: 0:00:14
[I 2022-10-11 22:18:12] ...trap/engines/engine.py.326:       process: 1.10503 | load: 0.00019
[I 2022-10-11 22:18:23] ...trap/engines/engine.py.321: eval: epoch 118 | batch 510/512
[I 2022-10-11 22:18:23] ...trap/engines/engine.py.325:       elapsed: 0:10:42 | left: 0:00:02
[I 2022-10-11 22:18:23] ...trap/engines/engine.py.326:       process: 1.10819 | load: 0.00015
[S 2022-10-11 22:18:26] ...trap/engines/engine.py.113: eval_epoch.epoch: 118
[I 2022-10-11 22:18:26] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 5 x 10000 matchs
[S 2022-10-11 22:22:06] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 2.00
[S 2022-10-11 22:22:06] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 2.00
[S 2022-10-11 22:22:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.478040
[S 2022-10-11 22:22:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.740640
[S 2022-10-11 22:22:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.818180
[S 2022-10-11 22:22:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.467400
[S 2022-10-11 22:22:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.733200
[S 2022-10-11 22:22:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.813100
[S 2022-10-11 22:22:06] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000
[S 2022-10-11 22:22:06] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000
[S 2022-10-11 22:22:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.005577
[S 2022-10-11 22:22:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.005292
[S 2022-10-11 22:22:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.003564
[S 2022-10-11 22:22:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.006082
[S 2022-10-11 22:22:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.006038
[S 2022-10-11 22:22:06] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.004146
[I 2022-10-11 22:23:20] ...trap/engines/engine.py.115: Ending evaluation procedures
[W 2022-10-11 22:23:22] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json'
[W 2022-10-11 22:23:22] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[W 2022-10-11 22:23:22] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[I 2022-10-11 22:23:29] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html
[I 2022-10-19 16:56:06] ...ytorch/bootstrap/run.py.73: Saving environment info
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75: options: 
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:   dataset:
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     aux_kw_path:
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       test: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_test_titles_kw.json
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       train: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_train_titles_kw.json
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       val: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_val_titles_kw.json
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     batch_sampler: triplet_classif
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     batch_size: 100
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     database: lmdb
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     dir: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     eval_split: test
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     freq_mismatch: 0.0
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     image_from: database
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     import: recipe1m.datasets.factory
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     kw_path:
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       test: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_test_ingr_kw.json
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       train: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_train_ingr_kw.json
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       val: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_val_ingr_kw.json
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     name: recipe1m
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     nb_classes: 1048
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     nb_threads: 4
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     randkw_p: 0.3
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     randkw_p_aux: 0.5
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     tokenized_raw_text: True
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     train_split: None
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     use_vcs: True
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:   engine:
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     name: logger
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     nb_epochs: 120
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     print_freq: 10
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     saving_criteria: ['eval_epoch.metric.med_im2recipe_mean:min', 'eval_epoch.metric.recall_at_1_im2recipe_mean:max']
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:   exp:
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     checkpoint: None
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     dir: /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     resume: best_eval_epoch.metric.recall_at_1_im2recipe_mean
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:   misc:
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     cuda: True
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     data_parrallel: False
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     logs_name: None
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     overrite: False
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     seed: 1338
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:   model:
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     criterion:
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:         tri_cross_encoder: False
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       itm_loss_weight: 1
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       keep_background: False
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       retrieval_strategy:
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:         aggregation: valid
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:         get_tokens: True
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:         margin: 0.05
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:         margin_params:
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:           increment: 0.005
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:           increment_margin: True
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:           max_margin: 0.3
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:         name: triplet
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:         nb_samples: 9999
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:         sampling: max_negative
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:         substrategy: ['IRR', 'RII', 'SIRR', 'SRII']
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:         substrategy_weights: [1.0, 1.0, 0.1, 0.1]
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     import: recipe1m.models.factory
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     metric:
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       k_test: 10
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       nb_bags: 2
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       nb_matchs_per_bag: 20000
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       retrieval_dir: /gpfsscratch/rech/dyf/ugz83ue/tmp/recipe1m
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       trijoint: True
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     name: trijoint
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     network:
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       activations: ['tanh', 'normalize']
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       aux_kwords: True
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       aux_kwords_encoder: True
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       avg_concat: True
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       bert_config: /linkhome/rech/genisi01/ugz83ue/tfood/recipe1m/models/networks/recipe_networks/config_bert_albef.json
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       cat_pos: 0
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       checkpoint: None
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       class_attention: False
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       class_attention_fus: False
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       cls_token: False
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       cls_token_fus: False
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       cross_decoder_image: True
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       cross_decoder_recipe: True
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:         class_attention: False
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:         cls_token: False
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:         cross_attention: False
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:         cross_decoder: True
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:         double_decoder: False
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:         get_tokens: False
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:         load_backbones: False
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:         n_heads: 4
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:         n_layers: 4
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:         query_img: False
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       cross_transformer: False
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       dim_emb: 1024
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       dim_image_out: 2048
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       dim_ingr_out: 300
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       dim_instr_in: 1024
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       dim_instr_out: 1024
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       embedding_names: ['proj_image']
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       freeze_im: False
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       freeze_rec: False
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       get_tokens: True
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       get_tokens_cross_decoder_recipe: True
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       hidden_size: 512
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       image_backbone_name: clip
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       image_res: 224
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       kwords_same_level: False
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       last_cat_aux: True
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       n_heads: 4
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       n_heads_cross: 4
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       n_heads_cross_image: 4
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       n_heads_single: 4
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       n_layers: 2
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       n_layers_cross: 2
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       n_layers_cross_image: 1
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       n_layers_single: 2
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       num_hidden_layers_kw: 2
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       old: True
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       path_ingrs: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/text/vocab.pkl
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       path_vocab: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/text/vocab_all.txt
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       recipe_encoder: tfoodvicha-transformer
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       text_encoder: bert-base-uncased
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       vision_width: 768
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       vit_all_tokens: True
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       with_ingrs: True
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       with_instrs: True
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:       with_titles: True
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     with_classif: False
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:   optimizer:
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     batch_size_factor: None
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     clip_grad: 8.0
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     import: recipe1m.optimizers.factory
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     lr: 1e-05
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     lr_img: 1e-06
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     name: trijoint_fixed_fine_tune
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:     switch_epoch: 20
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:   path_opts: /linkhome/rech/genisi01/ugz83ue/tfood/recipe1m/options/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_noimgproj_keepall.yaml
[S 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.75:   view: ['logs:train_epoch.loss', 'logs:train_epoch.bad_pairs', 'logs:eval_epoch.metric.med_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_1_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_5_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_10_im2recipe_mean', 'logs:eval_epoch.metric.med_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_1_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_5_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_10_recipe2im_mean', 'logs:optimizer.is_optimizer_recipe&image', 'logs:optimizer.total_norm']
[I 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.76: posix.uname_result(sysname='Linux', nodename='jean-zay-ia815', release='4.18.0-305.57.1.el8_4.x86_64', version='#1 SMP Tue Jul 5 17:55:55 EDT 2022', machine='x86_64')
[I 2022-10-19 16:56:12] ...ytorch/bootstrap/run.py.80: Available GPUs: [0]
[I 2022-10-19 16:56:12] ...strap/engines/factory.py.9: Creating engine...
[I 2022-10-19 16:56:12] ...trap/datasets/factory.py.8: Creating dataset...
[I 2022-10-19 16:56:25] ...m/datasets/recipe1m.py.152: Load VCs...
[I 2022-10-19 16:56:26] ...m/datasets/recipe1m.py.165: randkw_p... None
[I 2022-10-19 16:57:26] ...rap/datasets/factory.py.21: Evaluation will take place on testset (51303 items)
[I 2022-10-19 16:57:26] ...strap/models/factory.py.21: Creating model...
[I 2022-10-19 16:57:30] ...e_networks/networks.py.203: Loading CLIP Visual model...
[I 2022-10-19 16:57:34] ...e_networks/networks.py.145: cuda
[I 2022-10-19 16:57:35] ...networks/tfood_vicha.py.49: Load VCs encoder...
[I 2022-10-19 16:57:36] ...els/metrics/trijoint.py.46: self.save_ids False
[I 2022-10-19 16:57:36] ...strap/models/factory.py.44: Enabling CUDA mode...
[I 2022-10-19 16:57:43] ...etworks/tfood_vicha.py.184: training aux kw modules
[I 2022-10-19 16:57:43] ...etworks/tfood_vicha.py.184: training aux kw modules
[I 2022-10-19 16:57:43] ...otstrap/views/factory.py.8: Creating views...
[I 2022-10-19 16:57:47] ...strap/engines/engine.py.98: Loading best_eval_epoch.metric.recall_at_1_im2recipe_mean checkpoint
[I 2022-10-19 16:57:47] ...trap/engines/engine.py.440: Loading model...
[I 2022-10-19 16:57:54] ...trap/engines/engine.py.453: Loading engine...
[I 2022-10-19 16:57:54] ...trap/engines/engine.py.108: Launching evaluation procedures
[I 2022-10-19 16:57:54] ...trap/engines/engine.py.268: Evaluating model on testset for epoch 118
[I 2022-10-19 16:57:54] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler.
[I 2022-10-19 16:57:59] ...trap/engines/engine.py.321: eval: epoch 118 | batch 0/512
[I 2022-10-19 16:57:59] ...trap/engines/engine.py.325:       elapsed: 0:00:04 | left: 0:40:02
[I 2022-10-19 16:57:59] ...trap/engines/engine.py.326:       process: 4.69281 | load: 1.91455
[I 2022-10-19 16:58:11] ...trap/engines/engine.py.321: eval: epoch 118 | batch 10/512
[I 2022-10-19 16:58:11] ...trap/engines/engine.py.325:       elapsed: 0:00:17 | left: 0:13:35
[I 2022-10-19 16:58:11] ...trap/engines/engine.py.326:       process: 1.18180 | load: 0.00014
[I 2022-10-19 16:58:24] ...trap/engines/engine.py.321: eval: epoch 118 | batch 20/512
[I 2022-10-19 16:58:24] ...trap/engines/engine.py.325:       elapsed: 0:00:30 | left: 0:10:46
[I 2022-10-19 16:58:24] ...trap/engines/engine.py.326:       process: 1.30990 | load: 0.00014
[I 2022-10-19 16:58:36] ...trap/engines/engine.py.321: eval: epoch 118 | batch 30/512
[I 2022-10-19 16:58:36] ...trap/engines/engine.py.325:       elapsed: 0:00:41 | left: 0:09:44
[I 2022-10-19 16:58:36] ...trap/engines/engine.py.326:       process: 1.15990 | load: 0.00015
[I 2022-10-19 16:58:48] ...trap/engines/engine.py.321: eval: epoch 118 | batch 40/512
[I 2022-10-19 16:58:48] ...trap/engines/engine.py.325:       elapsed: 0:00:53 | left: 0:09:10
[I 2022-10-19 16:58:48] ...trap/engines/engine.py.326:       process: 1.14447 | load: 0.00015
[I 2022-10-19 16:59:00] ...trap/engines/engine.py.321: eval: epoch 118 | batch 50/512
[I 2022-10-19 16:59:00] ...trap/engines/engine.py.325:       elapsed: 0:01:05 | left: 0:09:11
[I 2022-10-19 16:59:00] ...trap/engines/engine.py.326:       process: 1.21604 | load: 0.00015
[I 2022-10-19 16:59:12] ...trap/engines/engine.py.321: eval: epoch 118 | batch 60/512
[I 2022-10-19 16:59:12] ...trap/engines/engine.py.325:       elapsed: 0:01:17 | left: 0:09:01
[I 2022-10-19 16:59:12] ...trap/engines/engine.py.326:       process: 1.14358 | load: 0.00016
[I 2022-10-19 16:59:24] ...trap/engines/engine.py.321: eval: epoch 118 | batch 70/512
[I 2022-10-19 16:59:24] ...trap/engines/engine.py.325:       elapsed: 0:01:30 | left: 0:09:17
[I 2022-10-19 16:59:24] ...trap/engines/engine.py.326:       process: 1.32220 | load: 0.00016
[I 2022-10-19 16:59:36] ...trap/engines/engine.py.321: eval: epoch 118 | batch 80/512
[I 2022-10-19 16:59:36] ...trap/engines/engine.py.325:       elapsed: 0:01:42 | left: 0:08:44
[I 2022-10-19 16:59:36] ...trap/engines/engine.py.326:       process: 1.24436 | load: 0.00015
[I 2022-10-19 16:59:48] ...trap/engines/engine.py.321: eval: epoch 118 | batch 90/512
[I 2022-10-19 16:59:48] ...trap/engines/engine.py.325:       elapsed: 0:01:54 | left: 0:08:25
[I 2022-10-19 16:59:48] ...trap/engines/engine.py.326:       process: 1.21833 | load: 0.00015
[I 2022-10-19 17:00:00] ...trap/engines/engine.py.321: eval: epoch 118 | batch 100/512
[I 2022-10-19 17:00:00] ...trap/engines/engine.py.325:       elapsed: 0:02:06 | left: 0:08:19
[I 2022-10-19 17:00:00] ...trap/engines/engine.py.326:       process: 1.19078 | load: 0.00015
[I 2022-10-19 17:00:12] ...trap/engines/engine.py.321: eval: epoch 118 | batch 110/512
[I 2022-10-19 17:00:12] ...trap/engines/engine.py.325:       elapsed: 0:02:18 | left: 0:08:01
[I 2022-10-19 17:00:12] ...trap/engines/engine.py.326:       process: 1.16830 | load: 0.00015
[I 2022-10-19 17:00:25] ...trap/engines/engine.py.321: eval: epoch 118 | batch 120/512
[I 2022-10-19 17:00:25] ...trap/engines/engine.py.325:       elapsed: 0:02:30 | left: 0:07:50
[I 2022-10-19 17:00:25] ...trap/engines/engine.py.326:       process: 1.16622 | load: 0.00015
[I 2022-10-19 17:00:36] ...trap/engines/engine.py.321: eval: epoch 118 | batch 130/512
[I 2022-10-19 17:00:36] ...trap/engines/engine.py.325:       elapsed: 0:02:42 | left: 0:07:34
[I 2022-10-19 17:00:36] ...trap/engines/engine.py.326:       process: 1.16521 | load: 0.00016
[I 2022-10-19 17:00:48] ...trap/engines/engine.py.321: eval: epoch 118 | batch 140/512
[I 2022-10-19 17:00:48] ...trap/engines/engine.py.325:       elapsed: 0:02:54 | left: 0:07:28
[I 2022-10-19 17:00:48] ...trap/engines/engine.py.326:       process: 1.25193 | load: 0.00015
[I 2022-10-19 17:01:00] ...trap/engines/engine.py.321: eval: epoch 118 | batch 150/512
[I 2022-10-19 17:01:00] ...trap/engines/engine.py.325:       elapsed: 0:03:06 | left: 0:07:10
[I 2022-10-19 17:01:00] ...trap/engines/engine.py.326:       process: 1.15787 | load: 0.00016
[I 2022-10-19 17:01:12] ...trap/engines/engine.py.321: eval: epoch 118 | batch 160/512
[I 2022-10-19 17:01:12] ...trap/engines/engine.py.325:       elapsed: 0:03:17 | left: 0:06:56
[I 2022-10-19 17:01:12] ...trap/engines/engine.py.326:       process: 1.19502 | load: 0.00016
[I 2022-10-19 17:01:24] ...trap/engines/engine.py.321: eval: epoch 118 | batch 170/512
[I 2022-10-19 17:01:24] ...trap/engines/engine.py.325:       elapsed: 0:03:29 | left: 0:06:45
[I 2022-10-19 17:01:24] ...trap/engines/engine.py.326:       process: 1.18197 | load: 0.00017
[I 2022-10-19 17:01:36] ...trap/engines/engine.py.321: eval: epoch 118 | batch 180/512
[I 2022-10-19 17:01:36] ...trap/engines/engine.py.325:       elapsed: 0:03:41 | left: 0:06:31
[I 2022-10-19 17:01:36] ...trap/engines/engine.py.326:       process: 1.16337 | load: 0.00016
[I 2022-10-19 17:01:48] ...trap/engines/engine.py.321: eval: epoch 118 | batch 190/512
[I 2022-10-19 17:01:48] ...trap/engines/engine.py.325:       elapsed: 0:03:53 | left: 0:06:17
[I 2022-10-19 17:01:48] ...trap/engines/engine.py.326:       process: 1.14644 | load: 0.00017
[I 2022-10-19 17:01:59] ...trap/engines/engine.py.321: eval: epoch 118 | batch 200/512
[I 2022-10-19 17:01:59] ...trap/engines/engine.py.325:       elapsed: 0:04:05 | left: 0:06:04
[I 2022-10-19 17:01:59] ...trap/engines/engine.py.326:       process: 1.15891 | load: 0.00017
[I 2022-10-19 17:02:11] ...trap/engines/engine.py.321: eval: epoch 118 | batch 210/512
[I 2022-10-19 17:02:11] ...trap/engines/engine.py.325:       elapsed: 0:04:16 | left: 0:05:53
[I 2022-10-19 17:02:11] ...trap/engines/engine.py.326:       process: 1.16563 | load: 0.00016
[I 2022-10-19 17:02:23] ...trap/engines/engine.py.321: eval: epoch 118 | batch 220/512
[I 2022-10-19 17:02:23] ...trap/engines/engine.py.325:       elapsed: 0:04:28 | left: 0:05:42
[I 2022-10-19 17:02:23] ...trap/engines/engine.py.326:       process: 1.17999 | load: 0.00016
[I 2022-10-19 17:02:34] ...trap/engines/engine.py.321: eval: epoch 118 | batch 230/512
[I 2022-10-19 17:02:34] ...trap/engines/engine.py.325:       elapsed: 0:04:40 | left: 0:05:29
[I 2022-10-19 17:02:34] ...trap/engines/engine.py.326:       process: 1.19185 | load: 0.00017
[I 2022-10-19 17:02:46] ...trap/engines/engine.py.321: eval: epoch 118 | batch 240/512
[I 2022-10-19 17:02:46] ...trap/engines/engine.py.325:       elapsed: 0:04:51 | left: 0:05:15
[I 2022-10-19 17:02:46] ...trap/engines/engine.py.326:       process: 1.16785 | load: 0.00016
[I 2022-10-19 17:02:58] ...trap/engines/engine.py.321: eval: epoch 118 | batch 250/512
[I 2022-10-19 17:02:58] ...trap/engines/engine.py.325:       elapsed: 0:05:03 | left: 0:05:06
[I 2022-10-19 17:02:58] ...trap/engines/engine.py.326:       process: 1.15297 | load: 0.00016
[I 2022-10-19 17:03:09] ...trap/engines/engine.py.321: eval: epoch 118 | batch 260/512
[I 2022-10-19 17:03:09] ...trap/engines/engine.py.325:       elapsed: 0:05:15 | left: 0:04:53
[I 2022-10-19 17:03:09] ...trap/engines/engine.py.326:       process: 1.16300 | load: 0.00017
[I 2022-10-19 17:03:21] ...trap/engines/engine.py.321: eval: epoch 118 | batch 270/512
[I 2022-10-19 17:03:21] ...trap/engines/engine.py.325:       elapsed: 0:05:26 | left: 0:04:40
[I 2022-10-19 17:03:21] ...trap/engines/engine.py.326:       process: 1.14022 | load: 0.00016
[I 2022-10-19 17:03:33] ...trap/engines/engine.py.321: eval: epoch 118 | batch 280/512
[I 2022-10-19 17:03:33] ...trap/engines/engine.py.325:       elapsed: 0:05:38 | left: 0:04:31
[I 2022-10-19 17:03:33] ...trap/engines/engine.py.326:       process: 1.17036 | load: 0.00016
[I 2022-10-19 17:03:44] ...trap/engines/engine.py.321: eval: epoch 118 | batch 290/512
[I 2022-10-19 17:03:44] ...trap/engines/engine.py.325:       elapsed: 0:05:50 | left: 0:04:20
[I 2022-10-19 17:03:44] ...trap/engines/engine.py.326:       process: 1.17383 | load: 0.00016
[I 2022-10-19 17:03:56] ...trap/engines/engine.py.321: eval: epoch 118 | batch 300/512
[I 2022-10-19 17:03:56] ...trap/engines/engine.py.325:       elapsed: 0:06:02 | left: 0:04:11
[I 2022-10-19 17:03:56] ...trap/engines/engine.py.326:       process: 1.18501 | load: 0.00016
[I 2022-10-19 17:04:08] ...trap/engines/engine.py.321: eval: epoch 118 | batch 310/512
[I 2022-10-19 17:04:08] ...trap/engines/engine.py.325:       elapsed: 0:06:13 | left: 0:03:58
[I 2022-10-19 17:04:08] ...trap/engines/engine.py.326:       process: 1.17938 | load: 0.00018
[I 2022-10-19 17:04:20] ...trap/engines/engine.py.321: eval: epoch 118 | batch 320/512
[I 2022-10-19 17:04:20] ...trap/engines/engine.py.325:       elapsed: 0:06:25 | left: 0:03:45
[I 2022-10-19 17:04:20] ...trap/engines/engine.py.326:       process: 1.15133 | load: 0.00015
[I 2022-10-19 17:04:32] ...trap/engines/engine.py.321: eval: epoch 118 | batch 330/512
[I 2022-10-19 17:04:32] ...trap/engines/engine.py.325:       elapsed: 0:06:37 | left: 0:03:39
[I 2022-10-19 17:04:32] ...trap/engines/engine.py.326:       process: 1.23994 | load: 0.00017
[I 2022-10-19 17:04:44] ...trap/engines/engine.py.321: eval: epoch 118 | batch 340/512
[I 2022-10-19 17:04:44] ...trap/engines/engine.py.325:       elapsed: 0:06:50 | left: 0:03:33
[I 2022-10-19 17:04:44] ...trap/engines/engine.py.326:       process: 1.24472 | load: 0.00018
[I 2022-10-19 17:04:57] ...trap/engines/engine.py.321: eval: epoch 118 | batch 350/512
[I 2022-10-19 17:04:57] ...trap/engines/engine.py.325:       elapsed: 0:07:02 | left: 0:03:18
[I 2022-10-19 17:04:57] ...trap/engines/engine.py.326:       process: 1.19960 | load: 0.00017
[I 2022-10-19 17:05:09] ...trap/engines/engine.py.321: eval: epoch 118 | batch 360/512
[I 2022-10-19 17:05:09] ...trap/engines/engine.py.325:       elapsed: 0:07:14 | left: 0:03:02
[I 2022-10-19 17:05:09] ...trap/engines/engine.py.326:       process: 1.20203 | load: 0.00018
[I 2022-10-19 17:05:21] ...trap/engines/engine.py.321: eval: epoch 118 | batch 370/512
[I 2022-10-19 17:05:21] ...trap/engines/engine.py.325:       elapsed: 0:07:26 | left: 0:02:49
[I 2022-10-19 17:05:21] ...trap/engines/engine.py.326:       process: 1.18394 | load: 0.00018
[I 2022-10-19 17:05:33] ...trap/engines/engine.py.321: eval: epoch 118 | batch 380/512
[I 2022-10-19 17:05:33] ...trap/engines/engine.py.325:       elapsed: 0:07:38 | left: 0:02:38
[I 2022-10-19 17:05:33] ...trap/engines/engine.py.326:       process: 1.18566 | load: 0.00017
[I 2022-10-19 17:05:45] ...trap/engines/engine.py.321: eval: epoch 118 | batch 390/512
[I 2022-10-19 17:05:45] ...trap/engines/engine.py.325:       elapsed: 0:07:50 | left: 0:02:27
[I 2022-10-19 17:05:45] ...trap/engines/engine.py.326:       process: 1.19995 | load: 0.00016
[I 2022-10-19 17:05:56] ...trap/engines/engine.py.321: eval: epoch 118 | batch 400/512
[I 2022-10-19 17:05:56] ...trap/engines/engine.py.325:       elapsed: 0:08:02 | left: 0:02:11
[I 2022-10-19 17:05:56] ...trap/engines/engine.py.326:       process: 1.06783 | load: 0.00018
[I 2022-10-19 17:06:07] ...trap/engines/engine.py.321: eval: epoch 118 | batch 410/512
[I 2022-10-19 17:06:07] ...trap/engines/engine.py.325:       elapsed: 0:08:13 | left: 0:01:51
[I 2022-10-19 17:06:07] ...trap/engines/engine.py.326:       process: 1.09185 | load: 0.00017
[I 2022-10-19 17:06:18] ...trap/engines/engine.py.321: eval: epoch 118 | batch 420/512
[I 2022-10-19 17:06:18] ...trap/engines/engine.py.325:       elapsed: 0:08:24 | left: 0:01:40
[I 2022-10-19 17:06:18] ...trap/engines/engine.py.326:       process: 1.10564 | load: 0.00016
[I 2022-10-19 17:06:29] ...trap/engines/engine.py.321: eval: epoch 118 | batch 430/512
[I 2022-10-19 17:06:29] ...trap/engines/engine.py.325:       elapsed: 0:08:34 | left: 0:01:28
[I 2022-10-19 17:06:29] ...trap/engines/engine.py.326:       process: 1.09452 | load: 0.00016
[I 2022-10-19 17:06:40] ...trap/engines/engine.py.321: eval: epoch 118 | batch 440/512
[I 2022-10-19 17:06:40] ...trap/engines/engine.py.325:       elapsed: 0:08:45 | left: 0:01:18
[I 2022-10-19 17:06:40] ...trap/engines/engine.py.326:       process: 1.09663 | load: 0.00016
[I 2022-10-19 17:06:51] ...trap/engines/engine.py.321: eval: epoch 118 | batch 450/512
[I 2022-10-19 17:06:51] ...trap/engines/engine.py.325:       elapsed: 0:08:56 | left: 0:01:07
[I 2022-10-19 17:06:51] ...trap/engines/engine.py.326:       process: 1.08639 | load: 0.00018
[I 2022-10-19 17:07:02] ...trap/engines/engine.py.321: eval: epoch 118 | batch 460/512
[I 2022-10-19 17:07:02] ...trap/engines/engine.py.325:       elapsed: 0:09:07 | left: 0:00:56
[I 2022-10-19 17:07:02] ...trap/engines/engine.py.326:       process: 1.08409 | load: 0.00017
[I 2022-10-19 17:07:13] ...trap/engines/engine.py.321: eval: epoch 118 | batch 470/512
[I 2022-10-19 17:07:13] ...trap/engines/engine.py.325:       elapsed: 0:09:18 | left: 0:00:45
[I 2022-10-19 17:07:13] ...trap/engines/engine.py.326:       process: 1.05964 | load: 0.00016
[I 2022-10-19 17:07:24] ...trap/engines/engine.py.321: eval: epoch 118 | batch 480/512
[I 2022-10-19 17:07:24] ...trap/engines/engine.py.325:       elapsed: 0:09:29 | left: 0:00:34
[I 2022-10-19 17:07:24] ...trap/engines/engine.py.326:       process: 1.07685 | load: 0.00016
[I 2022-10-19 17:07:34] ...trap/engines/engine.py.321: eval: epoch 118 | batch 490/512
[I 2022-10-19 17:07:34] ...trap/engines/engine.py.325:       elapsed: 0:09:40 | left: 0:00:23
[I 2022-10-19 17:07:34] ...trap/engines/engine.py.326:       process: 1.09865 | load: 0.00016
[I 2022-10-19 17:07:45] ...trap/engines/engine.py.321: eval: epoch 118 | batch 500/512
[I 2022-10-19 17:07:45] ...trap/engines/engine.py.325:       elapsed: 0:09:51 | left: 0:00:13
[I 2022-10-19 17:07:45] ...trap/engines/engine.py.326:       process: 1.09895 | load: 0.00016
[I 2022-10-19 17:07:56] ...trap/engines/engine.py.321: eval: epoch 118 | batch 510/512
[I 2022-10-19 17:07:56] ...trap/engines/engine.py.325:       elapsed: 0:10:02 | left: 0:00:02
[I 2022-10-19 17:07:56] ...trap/engines/engine.py.326:       process: 1.09538 | load: 0.00012
[S 2022-10-19 17:07:59] ...trap/engines/engine.py.113: eval_epoch.epoch: 118
[I 2022-10-19 17:07:59] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 2 x 20000 matchs
[S 2022-10-19 17:12:24] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 2.00
[S 2022-10-19 17:12:24] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 2.50
[S 2022-10-19 17:12:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.394875
[S 2022-10-19 17:12:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.657400
[S 2022-10-19 17:12:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.750250
[S 2022-10-19 17:12:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.381400
[S 2022-10-19 17:12:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.645100
[S 2022-10-19 17:12:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.741525
[S 2022-10-19 17:12:24] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000
[S 2022-10-19 17:12:24] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.500000
[S 2022-10-19 17:12:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.002725
[S 2022-10-19 17:12:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.001650
[S 2022-10-19 17:12:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.001350
[S 2022-10-19 17:12:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.002150
[S 2022-10-19 17:12:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.001300
[S 2022-10-19 17:12:24] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.000125
[I 2022-10-19 17:13:27] ...trap/engines/engine.py.115: Ending evaluation procedures
[W 2022-10-19 17:13:31] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json'
[W 2022-10-19 17:13:31] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[W 2022-10-19 17:13:31] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[I 2022-10-19 17:13:40] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html
[I 2022-10-19 17:15:44] ...ytorch/bootstrap/run.py.73: Saving environment info
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75: options: 
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:   dataset:
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     aux_kw_path:
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       test: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_test_titles_kw.json
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       train: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_train_titles_kw.json
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       val: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_val_titles_kw.json
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     batch_sampler: triplet_classif
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     batch_size: 100
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     database: lmdb
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     dir: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     eval_split: test
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     freq_mismatch: 0.0
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     image_from: database
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     import: recipe1m.datasets.factory
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     kw_path:
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       test: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_test_ingr_kw.json
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       train: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_train_ingr_kw.json
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       val: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_val_ingr_kw.json
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     name: recipe1m
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     nb_classes: 1048
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     nb_threads: 4
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     randkw_p: 0.3
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     randkw_p_aux: 0.5
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     tokenized_raw_text: True
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     train_split: None
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     use_vcs: True
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:   engine:
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     name: logger
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     nb_epochs: 120
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     print_freq: 10
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     saving_criteria: ['eval_epoch.metric.med_im2recipe_mean:min', 'eval_epoch.metric.recall_at_1_im2recipe_mean:max']
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:   exp:
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     checkpoint: None
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     dir: /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     resume: best_eval_epoch.metric.recall_at_1_im2recipe_mean
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:   misc:
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     cuda: True
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     data_parrallel: False
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     logs_name: None
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     overrite: False
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     seed: 1338
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:   model:
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     criterion:
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:         tri_cross_encoder: False
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       itm_loss_weight: 1
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       keep_background: False
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       retrieval_strategy:
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:         aggregation: valid
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:         get_tokens: True
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:         margin: 0.05
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:         margin_params:
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:           increment: 0.005
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:           increment_margin: True
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:           max_margin: 0.3
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:         name: triplet
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:         nb_samples: 9999
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:         sampling: max_negative
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:         substrategy: ['IRR', 'RII', 'SIRR', 'SRII']
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:         substrategy_weights: [1.0, 1.0, 0.1, 0.1]
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     import: recipe1m.models.factory
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     metric:
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       k_test: 10
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       nb_bags: 1
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       nb_matchs_per_bag: 30000
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       retrieval_dir: /gpfsscratch/rech/dyf/ugz83ue/tmp/recipe1m
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       trijoint: True
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     name: trijoint
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     network:
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       activations: ['tanh', 'normalize']
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       aux_kwords: True
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       aux_kwords_encoder: True
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       avg_concat: True
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       bert_config: /linkhome/rech/genisi01/ugz83ue/tfood/recipe1m/models/networks/recipe_networks/config_bert_albef.json
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       cat_pos: 0
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       checkpoint: None
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       class_attention: False
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       class_attention_fus: False
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       cls_token: False
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       cls_token_fus: False
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       cross_decoder_image: True
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       cross_decoder_recipe: True
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:         class_attention: False
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:         cls_token: False
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:         cross_attention: False
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:         cross_decoder: True
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:         double_decoder: False
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:         get_tokens: False
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:         load_backbones: False
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:         n_heads: 4
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:         n_layers: 4
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:         query_img: False
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       cross_transformer: False
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       dim_emb: 1024
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       dim_image_out: 2048
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       dim_ingr_out: 300
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       dim_instr_in: 1024
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       dim_instr_out: 1024
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       embedding_names: ['proj_image']
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       freeze_im: False
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       freeze_rec: False
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       get_tokens: True
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       get_tokens_cross_decoder_recipe: True
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       hidden_size: 512
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       image_backbone_name: clip
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       image_res: 224
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       kwords_same_level: False
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       last_cat_aux: True
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       n_heads: 4
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       n_heads_cross: 4
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       n_heads_cross_image: 4
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       n_heads_single: 4
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       n_layers: 2
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       n_layers_cross: 2
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       n_layers_cross_image: 1
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       n_layers_single: 2
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       num_hidden_layers_kw: 2
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       old: True
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       path_ingrs: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/text/vocab.pkl
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       path_vocab: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/text/vocab_all.txt
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       recipe_encoder: tfoodvicha-transformer
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       text_encoder: bert-base-uncased
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       vision_width: 768
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       vit_all_tokens: True
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       with_ingrs: True
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       with_instrs: True
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:       with_titles: True
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     with_classif: False
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:   optimizer:
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     batch_size_factor: None
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     clip_grad: 8.0
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     import: recipe1m.optimizers.factory
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     lr: 1e-05
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     lr_img: 1e-06
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     name: trijoint_fixed_fine_tune
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:     switch_epoch: 20
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:   path_opts: /linkhome/rech/genisi01/ugz83ue/tfood/recipe1m/options/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_noimgproj_keepall.yaml
[S 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.75:   view: ['logs:train_epoch.loss', 'logs:train_epoch.bad_pairs', 'logs:eval_epoch.metric.med_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_1_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_5_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_10_im2recipe_mean', 'logs:eval_epoch.metric.med_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_1_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_5_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_10_recipe2im_mean', 'logs:optimizer.is_optimizer_recipe&image', 'logs:optimizer.total_norm']
[I 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.76: posix.uname_result(sysname='Linux', nodename='jean-zay-ia815', release='4.18.0-305.57.1.el8_4.x86_64', version='#1 SMP Tue Jul 5 17:55:55 EDT 2022', machine='x86_64')
[I 2022-10-19 17:15:51] ...ytorch/bootstrap/run.py.80: Available GPUs: [0]
[I 2022-10-19 17:15:51] ...strap/engines/factory.py.9: Creating engine...
[I 2022-10-19 17:15:51] ...trap/datasets/factory.py.8: Creating dataset...
[I 2022-10-19 17:16:06] ...m/datasets/recipe1m.py.152: Load VCs...
[I 2022-10-19 17:16:06] ...m/datasets/recipe1m.py.165: randkw_p... None
[I 2022-10-19 17:17:07] ...rap/datasets/factory.py.21: Evaluation will take place on testset (51303 items)
[I 2022-10-19 17:17:07] ...strap/models/factory.py.21: Creating model...
[I 2022-10-19 17:17:11] ...e_networks/networks.py.203: Loading CLIP Visual model...
[I 2022-10-19 17:17:15] ...e_networks/networks.py.145: cuda
[I 2022-10-19 17:17:16] ...networks/tfood_vicha.py.49: Load VCs encoder...
[I 2022-10-19 17:17:17] ...els/metrics/trijoint.py.46: self.save_ids False
[I 2022-10-19 17:17:17] ...strap/models/factory.py.44: Enabling CUDA mode...
[I 2022-10-19 17:17:20] ...etworks/tfood_vicha.py.184: training aux kw modules
[I 2022-10-19 17:17:20] ...etworks/tfood_vicha.py.184: training aux kw modules
[I 2022-10-19 17:17:20] ...otstrap/views/factory.py.8: Creating views...
[I 2022-10-19 17:17:23] ...strap/engines/engine.py.98: Loading best_eval_epoch.metric.recall_at_1_im2recipe_mean checkpoint
[I 2022-10-19 17:17:23] ...trap/engines/engine.py.440: Loading model...
[I 2022-10-19 17:17:29] ...trap/engines/engine.py.453: Loading engine...
[I 2022-10-19 17:17:29] ...trap/engines/engine.py.108: Launching evaluation procedures
[I 2022-10-19 17:17:29] ...trap/engines/engine.py.268: Evaluating model on testset for epoch 118
[I 2022-10-19 17:17:29] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler.
[I 2022-10-19 17:17:33] ...trap/engines/engine.py.321: eval: epoch 118 | batch 0/512
[I 2022-10-19 17:17:33] ...trap/engines/engine.py.325:       elapsed: 0:00:03 | left: 0:26:59
[I 2022-10-19 17:17:33] ...trap/engines/engine.py.326:       process: 3.16260 | load: 1.18795
[I 2022-10-19 17:17:47] ...trap/engines/engine.py.321: eval: epoch 118 | batch 10/512
[I 2022-10-19 17:17:47] ...trap/engines/engine.py.325:       elapsed: 0:00:17 | left: 0:13:24
[I 2022-10-19 17:17:47] ...trap/engines/engine.py.326:       process: 1.34923 | load: 0.00014
[I 2022-10-19 17:18:00] ...trap/engines/engine.py.321: eval: epoch 118 | batch 20/512
[I 2022-10-19 17:18:00] ...trap/engines/engine.py.325:       elapsed: 0:00:30 | left: 0:11:17
[I 2022-10-19 17:18:00] ...trap/engines/engine.py.326:       process: 1.34699 | load: 0.00015
[I 2022-10-19 17:18:14] ...trap/engines/engine.py.321: eval: epoch 118 | batch 30/512
[I 2022-10-19 17:18:14] ...trap/engines/engine.py.325:       elapsed: 0:00:44 | left: 0:10:48
[I 2022-10-19 17:18:14] ...trap/engines/engine.py.326:       process: 1.27097 | load: 0.00014
[I 2022-10-19 17:18:26] ...trap/engines/engine.py.321: eval: epoch 118 | batch 40/512
[I 2022-10-19 17:18:26] ...trap/engines/engine.py.325:       elapsed: 0:00:56 | left: 0:09:59
[I 2022-10-19 17:18:26] ...trap/engines/engine.py.326:       process: 1.27278 | load: 0.00015
[I 2022-10-19 17:18:38] ...trap/engines/engine.py.321: eval: epoch 118 | batch 50/512
[I 2022-10-19 17:18:38] ...trap/engines/engine.py.325:       elapsed: 0:01:08 | left: 0:09:19
[I 2022-10-19 17:18:38] ...trap/engines/engine.py.326:       process: 1.21728 | load: 0.00014
[I 2022-10-19 17:18:50] ...trap/engines/engine.py.321: eval: epoch 118 | batch 60/512
[I 2022-10-19 17:18:50] ...trap/engines/engine.py.325:       elapsed: 0:01:20 | left: 0:08:54
[I 2022-10-19 17:18:50] ...trap/engines/engine.py.326:       process: 1.15743 | load: 0.00015
[I 2022-10-19 17:19:02] ...trap/engines/engine.py.321: eval: epoch 118 | batch 70/512
[I 2022-10-19 17:19:02] ...trap/engines/engine.py.325:       elapsed: 0:01:32 | left: 0:09:06
[I 2022-10-19 17:19:02] ...trap/engines/engine.py.326:       process: 1.38702 | load: 0.00016
[I 2022-10-19 17:19:14] ...trap/engines/engine.py.321: eval: epoch 118 | batch 80/512
[I 2022-10-19 17:19:14] ...trap/engines/engine.py.325:       elapsed: 0:01:44 | left: 0:08:31
[I 2022-10-19 17:19:14] ...trap/engines/engine.py.326:       process: 1.17989 | load: 0.00016
[I 2022-10-19 17:19:26] ...trap/engines/engine.py.321: eval: epoch 118 | batch 90/512
[I 2022-10-19 17:19:26] ...trap/engines/engine.py.325:       elapsed: 0:01:56 | left: 0:08:30
[I 2022-10-19 17:19:26] ...trap/engines/engine.py.326:       process: 1.29606 | load: 0.00015
[I 2022-10-19 17:19:38] ...trap/engines/engine.py.321: eval: epoch 118 | batch 100/512
[I 2022-10-19 17:19:38] ...trap/engines/engine.py.325:       elapsed: 0:02:08 | left: 0:08:05
[I 2022-10-19 17:19:38] ...trap/engines/engine.py.326:       process: 1.17552 | load: 0.00016
[I 2022-10-19 17:19:49] ...trap/engines/engine.py.321: eval: epoch 118 | batch 110/512
[I 2022-10-19 17:19:49] ...trap/engines/engine.py.325:       elapsed: 0:02:19 | left: 0:07:47
[I 2022-10-19 17:19:49] ...trap/engines/engine.py.326:       process: 1.16510 | load: 0.00016
[I 2022-10-19 17:20:01] ...trap/engines/engine.py.321: eval: epoch 118 | batch 120/512
[I 2022-10-19 17:20:01] ...trap/engines/engine.py.325:       elapsed: 0:02:31 | left: 0:07:47
[I 2022-10-19 17:20:01] ...trap/engines/engine.py.326:       process: 1.21547 | load: 0.00016
[I 2022-10-19 17:20:13] ...trap/engines/engine.py.321: eval: epoch 118 | batch 130/512
[I 2022-10-19 17:20:13] ...trap/engines/engine.py.325:       elapsed: 0:02:43 | left: 0:07:28
[I 2022-10-19 17:20:13] ...trap/engines/engine.py.326:       process: 1.15423 | load: 0.00016
[I 2022-10-19 17:20:25] ...trap/engines/engine.py.321: eval: epoch 118 | batch 140/512
[I 2022-10-19 17:20:25] ...trap/engines/engine.py.325:       elapsed: 0:02:55 | left: 0:07:19
[I 2022-10-19 17:20:25] ...trap/engines/engine.py.326:       process: 1.19237 | load: 0.00017
[I 2022-10-19 17:20:37] ...trap/engines/engine.py.321: eval: epoch 118 | batch 150/512
[I 2022-10-19 17:20:37] ...trap/engines/engine.py.325:       elapsed: 0:03:07 | left: 0:07:12
[I 2022-10-19 17:20:37] ...trap/engines/engine.py.326:       process: 1.18057 | load: 0.00016
[I 2022-10-19 17:20:48] ...trap/engines/engine.py.321: eval: epoch 118 | batch 160/512
[I 2022-10-19 17:20:48] ...trap/engines/engine.py.325:       elapsed: 0:03:18 | left: 0:06:57
[I 2022-10-19 17:20:48] ...trap/engines/engine.py.326:       process: 1.17712 | load: 0.00016
[I 2022-10-19 17:21:00] ...trap/engines/engine.py.321: eval: epoch 118 | batch 170/512
[I 2022-10-19 17:21:00] ...trap/engines/engine.py.325:       elapsed: 0:03:30 | left: 0:06:43
[I 2022-10-19 17:21:00] ...trap/engines/engine.py.326:       process: 1.17814 | load: 0.00016
[I 2022-10-19 17:21:12] ...trap/engines/engine.py.321: eval: epoch 118 | batch 180/512
[I 2022-10-19 17:21:12] ...trap/engines/engine.py.325:       elapsed: 0:03:42 | left: 0:06:31
[I 2022-10-19 17:21:12] ...trap/engines/engine.py.326:       process: 1.15775 | load: 0.00015
[I 2022-10-19 17:21:24] ...trap/engines/engine.py.321: eval: epoch 118 | batch 190/512
[I 2022-10-19 17:21:24] ...trap/engines/engine.py.325:       elapsed: 0:03:54 | left: 0:06:20
[I 2022-10-19 17:21:24] ...trap/engines/engine.py.326:       process: 1.14839 | load: 0.00016
[I 2022-10-19 17:21:36] ...trap/engines/engine.py.321: eval: epoch 118 | batch 200/512
[I 2022-10-19 17:21:36] ...trap/engines/engine.py.325:       elapsed: 0:04:06 | left: 0:06:05
[I 2022-10-19 17:21:36] ...trap/engines/engine.py.326:       process: 1.15738 | load: 0.00016
[I 2022-10-19 17:21:47] ...trap/engines/engine.py.321: eval: epoch 118 | batch 210/512
[I 2022-10-19 17:21:47] ...trap/engines/engine.py.325:       elapsed: 0:04:17 | left: 0:05:48
[I 2022-10-19 17:21:47] ...trap/engines/engine.py.326:       process: 1.14655 | load: 0.00016
[I 2022-10-19 17:21:59] ...trap/engines/engine.py.321: eval: epoch 118 | batch 220/512
[I 2022-10-19 17:21:59] ...trap/engines/engine.py.325:       elapsed: 0:04:29 | left: 0:05:41
[I 2022-10-19 17:21:59] ...trap/engines/engine.py.326:       process: 1.20834 | load: 0.00017
[I 2022-10-19 17:22:10] ...trap/engines/engine.py.321: eval: epoch 118 | batch 230/512
[I 2022-10-19 17:22:10] ...trap/engines/engine.py.325:       elapsed: 0:04:41 | left: 0:05:28
[I 2022-10-19 17:22:11] ...trap/engines/engine.py.326:       process: 1.14794 | load: 0.00016
[I 2022-10-19 17:22:22] ...trap/engines/engine.py.321: eval: epoch 118 | batch 240/512
[I 2022-10-19 17:22:22] ...trap/engines/engine.py.325:       elapsed: 0:04:52 | left: 0:05:17
[I 2022-10-19 17:22:22] ...trap/engines/engine.py.326:       process: 1.15896 | load: 0.00017
[I 2022-10-19 17:22:34] ...trap/engines/engine.py.321: eval: epoch 118 | batch 250/512
[I 2022-10-19 17:22:34] ...trap/engines/engine.py.325:       elapsed: 0:05:04 | left: 0:05:04
[I 2022-10-19 17:22:34] ...trap/engines/engine.py.326:       process: 1.15383 | load: 0.00016
[I 2022-10-19 17:22:45] ...trap/engines/engine.py.321: eval: epoch 118 | batch 260/512
[I 2022-10-19 17:22:45] ...trap/engines/engine.py.325:       elapsed: 0:05:15 | left: 0:04:51
[I 2022-10-19 17:22:45] ...trap/engines/engine.py.326:       process: 1.15848 | load: 0.00017
[I 2022-10-19 17:22:57] ...trap/engines/engine.py.321: eval: epoch 118 | batch 270/512
[I 2022-10-19 17:22:57] ...trap/engines/engine.py.325:       elapsed: 0:05:27 | left: 0:04:39
[I 2022-10-19 17:22:57] ...trap/engines/engine.py.326:       process: 1.12536 | load: 0.00016
[I 2022-10-19 17:23:09] ...trap/engines/engine.py.321: eval: epoch 118 | batch 280/512
[I 2022-10-19 17:23:09] ...trap/engines/engine.py.325:       elapsed: 0:05:39 | left: 0:04:30
[I 2022-10-19 17:23:09] ...trap/engines/engine.py.326:       process: 1.16756 | load: 0.00016
[I 2022-10-19 17:23:20] ...trap/engines/engine.py.321: eval: epoch 118 | batch 290/512
[I 2022-10-19 17:23:20] ...trap/engines/engine.py.325:       elapsed: 0:05:50 | left: 0:04:16
[I 2022-10-19 17:23:20] ...trap/engines/engine.py.326:       process: 1.15663 | load: 0.00016
[I 2022-10-19 17:23:32] ...trap/engines/engine.py.321: eval: epoch 118 | batch 300/512
[I 2022-10-19 17:23:32] ...trap/engines/engine.py.325:       elapsed: 0:06:02 | left: 0:04:02
[I 2022-10-19 17:23:32] ...trap/engines/engine.py.326:       process: 1.09156 | load: 0.00017
[I 2022-10-19 17:23:43] ...trap/engines/engine.py.321: eval: epoch 118 | batch 310/512
[I 2022-10-19 17:23:43] ...trap/engines/engine.py.325:       elapsed: 0:06:13 | left: 0:03:41
[I 2022-10-19 17:23:43] ...trap/engines/engine.py.326:       process: 1.10035 | load: 0.00016
[I 2022-10-19 17:23:53] ...trap/engines/engine.py.321: eval: epoch 118 | batch 320/512
[I 2022-10-19 17:23:53] ...trap/engines/engine.py.325:       elapsed: 0:06:23 | left: 0:03:27
[I 2022-10-19 17:23:53] ...trap/engines/engine.py.326:       process: 1.06782 | load: 0.00016
[I 2022-10-19 17:24:04] ...trap/engines/engine.py.321: eval: epoch 118 | batch 330/512
[I 2022-10-19 17:24:04] ...trap/engines/engine.py.325:       elapsed: 0:06:34 | left: 0:03:18
[I 2022-10-19 17:24:04] ...trap/engines/engine.py.326:       process: 1.09017 | load: 0.00015
[I 2022-10-19 17:24:15] ...trap/engines/engine.py.321: eval: epoch 118 | batch 340/512
[I 2022-10-19 17:24:15] ...trap/engines/engine.py.325:       elapsed: 0:06:45 | left: 0:03:07
[I 2022-10-19 17:24:15] ...trap/engines/engine.py.326:       process: 1.08301 | load: 0.00016
[I 2022-10-19 17:24:26] ...trap/engines/engine.py.321: eval: epoch 118 | batch 350/512
[I 2022-10-19 17:24:26] ...trap/engines/engine.py.325:       elapsed: 0:06:56 | left: 0:02:56
[I 2022-10-19 17:24:26] ...trap/engines/engine.py.326:       process: 1.07725 | load: 0.00016
[I 2022-10-19 17:24:37] ...trap/engines/engine.py.321: eval: epoch 118 | batch 360/512
[I 2022-10-19 17:24:37] ...trap/engines/engine.py.325:       elapsed: 0:07:07 | left: 0:02:46
[I 2022-10-19 17:24:37] ...trap/engines/engine.py.326:       process: 1.09737 | load: 0.00015
[I 2022-10-19 17:24:48] ...trap/engines/engine.py.321: eval: epoch 118 | batch 370/512
[I 2022-10-19 17:24:48] ...trap/engines/engine.py.325:       elapsed: 0:07:18 | left: 0:02:34
[I 2022-10-19 17:24:48] ...trap/engines/engine.py.326:       process: 1.07621 | load: 0.00015
[I 2022-10-19 17:24:59] ...trap/engines/engine.py.321: eval: epoch 118 | batch 380/512
[I 2022-10-19 17:24:59] ...trap/engines/engine.py.325:       elapsed: 0:07:29 | left: 0:02:23
[I 2022-10-19 17:24:59] ...trap/engines/engine.py.326:       process: 1.08783 | load: 0.00016
[I 2022-10-19 17:25:10] ...trap/engines/engine.py.321: eval: epoch 118 | batch 390/512
[I 2022-10-19 17:25:10] ...trap/engines/engine.py.325:       elapsed: 0:07:40 | left: 0:02:13
[I 2022-10-19 17:25:10] ...trap/engines/engine.py.326:       process: 1.09188 | load: 0.00016
[I 2022-10-19 17:25:21] ...trap/engines/engine.py.321: eval: epoch 118 | batch 400/512
[I 2022-10-19 17:25:21] ...trap/engines/engine.py.325:       elapsed: 0:07:51 | left: 0:02:00
[I 2022-10-19 17:25:21] ...trap/engines/engine.py.326:       process: 1.06280 | load: 0.00016
[I 2022-10-19 17:25:31] ...trap/engines/engine.py.321: eval: epoch 118 | batch 410/512
[I 2022-10-19 17:25:31] ...trap/engines/engine.py.325:       elapsed: 0:08:01 | left: 0:01:50
[I 2022-10-19 17:25:31] ...trap/engines/engine.py.326:       process: 1.09224 | load: 0.00016
[I 2022-10-19 17:25:42] ...trap/engines/engine.py.321: eval: epoch 118 | batch 420/512
[I 2022-10-19 17:25:42] ...trap/engines/engine.py.325:       elapsed: 0:08:12 | left: 0:01:40
[I 2022-10-19 17:25:42] ...trap/engines/engine.py.326:       process: 1.10959 | load: 0.00015
[I 2022-10-19 17:25:53] ...trap/engines/engine.py.321: eval: epoch 118 | batch 430/512
[I 2022-10-19 17:25:53] ...trap/engines/engine.py.325:       elapsed: 0:08:23 | left: 0:01:28
[I 2022-10-19 17:25:53] ...trap/engines/engine.py.326:       process: 1.10209 | load: 0.00016
[I 2022-10-19 17:26:04] ...trap/engines/engine.py.321: eval: epoch 118 | batch 440/512
[I 2022-10-19 17:26:04] ...trap/engines/engine.py.325:       elapsed: 0:08:34 | left: 0:01:18
[I 2022-10-19 17:26:04] ...trap/engines/engine.py.326:       process: 1.09459 | load: 0.00016
[I 2022-10-19 17:26:15] ...trap/engines/engine.py.321: eval: epoch 118 | batch 450/512
[I 2022-10-19 17:26:15] ...trap/engines/engine.py.325:       elapsed: 0:08:45 | left: 0:01:07
[I 2022-10-19 17:26:15] ...trap/engines/engine.py.326:       process: 1.08823 | load: 0.00015
[I 2022-10-19 17:26:26] ...trap/engines/engine.py.321: eval: epoch 118 | batch 460/512
[I 2022-10-19 17:26:26] ...trap/engines/engine.py.325:       elapsed: 0:08:56 | left: 0:00:56
[I 2022-10-19 17:26:26] ...trap/engines/engine.py.326:       process: 1.08545 | load: 0.00016
[I 2022-10-19 17:26:36] ...trap/engines/engine.py.321: eval: epoch 118 | batch 470/512
[I 2022-10-19 17:26:36] ...trap/engines/engine.py.325:       elapsed: 0:09:06 | left: 0:00:45
[I 2022-10-19 17:26:36] ...trap/engines/engine.py.326:       process: 1.06151 | load: 0.00016
[I 2022-10-19 17:26:47] ...trap/engines/engine.py.321: eval: epoch 118 | batch 480/512
[I 2022-10-19 17:26:47] ...trap/engines/engine.py.325:       elapsed: 0:09:17 | left: 0:00:34
[I 2022-10-19 17:26:47] ...trap/engines/engine.py.326:       process: 1.07887 | load: 0.00015
[I 2022-10-19 17:26:58] ...trap/engines/engine.py.321: eval: epoch 118 | batch 490/512
[I 2022-10-19 17:26:58] ...trap/engines/engine.py.325:       elapsed: 0:09:28 | left: 0:00:23
[I 2022-10-19 17:26:58] ...trap/engines/engine.py.326:       process: 1.09534 | load: 0.00015
[I 2022-10-19 17:27:09] ...trap/engines/engine.py.321: eval: epoch 118 | batch 500/512
[I 2022-10-19 17:27:09] ...trap/engines/engine.py.325:       elapsed: 0:09:39 | left: 0:00:13
[I 2022-10-19 17:27:09] ...trap/engines/engine.py.326:       process: 1.09412 | load: 0.00016
[I 2022-10-19 17:27:20] ...trap/engines/engine.py.321: eval: epoch 118 | batch 510/512
[I 2022-10-19 17:27:20] ...trap/engines/engine.py.325:       elapsed: 0:09:50 | left: 0:00:02
[I 2022-10-19 17:27:20] ...trap/engines/engine.py.326:       process: 1.09364 | load: 0.00010
[S 2022-10-19 17:27:22] ...trap/engines/engine.py.113: eval_epoch.epoch: 118
[I 2022-10-19 17:27:22] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 1 x 30000 matchs
[S 2022-10-19 17:31:51] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 3.00
[S 2022-10-19 17:31:51] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 3.00
[S 2022-10-19 17:31:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.350700
[S 2022-10-19 17:31:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.601033
[S 2022-10-19 17:31:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.702200
[S 2022-10-19 17:31:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.334967
[S 2022-10-19 17:31:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.591967
[S 2022-10-19 17:31:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.691000
[S 2022-10-19 17:31:51] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000
[S 2022-10-19 17:31:51] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000
[S 2022-10-19 17:31:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.000000
[S 2022-10-19 17:31:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.000000
[S 2022-10-19 17:31:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.000000
[S 2022-10-19 17:31:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.000000
[S 2022-10-19 17:31:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.000000
[S 2022-10-19 17:31:51] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.000000
[I 2022-10-19 17:32:38] ...trap/engines/engine.py.115: Ending evaluation procedures
[W 2022-10-19 17:32:42] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json'
[W 2022-10-19 17:32:42] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[W 2022-10-19 17:32:42] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[I 2022-10-19 17:32:51] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html
[I 2022-10-19 17:58:30] ...ytorch/bootstrap/run.py.73: Saving environment info
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75: options: 
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:   dataset:
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     aux_kw_path:
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       test: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_test_titles_kw.json
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       train: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_train_titles_kw.json
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       val: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_val_titles_kw.json
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     batch_sampler: triplet_classif
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     batch_size: 100
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     database: lmdb
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     dir: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     eval_split: test
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     freq_mismatch: 0.0
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     image_from: database
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     import: recipe1m.datasets.factory
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     kw_path:
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       test: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_test_ingr_kw.json
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       train: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_train_ingr_kw.json
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       val: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_val_ingr_kw.json
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     name: recipe1m
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     nb_classes: 1048
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     nb_threads: 4
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     randkw_p: 0.3
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     randkw_p_aux: 0.5
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     tokenized_raw_text: True
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     train_split: None
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     use_vcs: True
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:   engine:
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     name: logger
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     nb_epochs: 120
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     print_freq: 10
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     saving_criteria: ['eval_epoch.metric.med_im2recipe_mean:min', 'eval_epoch.metric.recall_at_1_im2recipe_mean:max']
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:   exp:
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     checkpoint: None
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     dir: /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     resume: best_eval_epoch.metric.recall_at_1_im2recipe_mean
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:   misc:
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     cuda: True
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     data_parrallel: False
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     logs_name: None
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     overrite: False
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     seed: 1338
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:   model:
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     criterion:
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:         tri_cross_encoder: False
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       itm_loss_weight: 1
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       keep_background: False
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       retrieval_strategy:
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:         aggregation: valid
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:         get_tokens: True
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:         margin: 0.05
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:         margin_params:
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:           increment: 0.005
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:           increment_margin: True
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:           max_margin: 0.3
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:         name: triplet
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:         nb_samples: 9999
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:         sampling: max_negative
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:         substrategy: ['IRR', 'RII', 'SIRR', 'SRII']
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:         substrategy_weights: [1.0, 1.0, 0.1, 0.1]
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     import: recipe1m.models.factory
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     metric:
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       k_test: 10
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       nb_bags: 1
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       nb_matchs_per_bag: 40000
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       retrieval_dir: /gpfsscratch/rech/dyf/ugz83ue/tmp/recipe1m
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       trijoint: True
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     name: trijoint
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     network:
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       activations: ['tanh', 'normalize']
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       aux_kwords: True
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       aux_kwords_encoder: True
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       avg_concat: True
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       bert_config: /linkhome/rech/genisi01/ugz83ue/tfood/recipe1m/models/networks/recipe_networks/config_bert_albef.json
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       cat_pos: 0
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       checkpoint: None
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       class_attention: False
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       class_attention_fus: False
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       cls_token: False
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       cls_token_fus: False
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       cross_decoder_image: True
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       cross_decoder_recipe: True
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:         class_attention: False
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:         cls_token: False
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:         cross_attention: False
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:         cross_decoder: True
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:         double_decoder: False
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:         get_tokens: False
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:         load_backbones: False
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:         n_heads: 4
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:         n_layers: 4
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:         query_img: False
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       cross_transformer: False
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       dim_emb: 1024
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       dim_image_out: 2048
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       dim_ingr_out: 300
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       dim_instr_in: 1024
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       dim_instr_out: 1024
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       embedding_names: ['proj_image']
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       freeze_im: False
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       freeze_rec: False
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       get_tokens: True
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       get_tokens_cross_decoder_recipe: True
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       hidden_size: 512
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       image_backbone_name: clip
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       image_res: 224
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       kwords_same_level: False
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       last_cat_aux: True
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       n_heads: 4
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       n_heads_cross: 4
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       n_heads_cross_image: 4
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       n_heads_single: 4
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       n_layers: 2
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       n_layers_cross: 2
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       n_layers_cross_image: 1
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       n_layers_single: 2
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       num_hidden_layers_kw: 2
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       old: True
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       path_ingrs: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/text/vocab.pkl
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       path_vocab: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/text/vocab_all.txt
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       recipe_encoder: tfoodvicha-transformer
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       text_encoder: bert-base-uncased
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       vision_width: 768
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       vit_all_tokens: True
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       with_ingrs: True
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       with_instrs: True
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:       with_titles: True
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     with_classif: False
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:   optimizer:
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     batch_size_factor: None
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     clip_grad: 8.0
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     import: recipe1m.optimizers.factory
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     lr: 1e-05
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     lr_img: 1e-06
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     name: trijoint_fixed_fine_tune
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:     switch_epoch: 20
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:   path_opts: /linkhome/rech/genisi01/ugz83ue/tfood/recipe1m/options/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_noimgproj_keepall.yaml
[S 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.75:   view: ['logs:train_epoch.loss', 'logs:train_epoch.bad_pairs', 'logs:eval_epoch.metric.med_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_1_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_5_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_10_im2recipe_mean', 'logs:eval_epoch.metric.med_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_1_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_5_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_10_recipe2im_mean', 'logs:optimizer.is_optimizer_recipe&image', 'logs:optimizer.total_norm']
[I 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.76: posix.uname_result(sysname='Linux', nodename='jean-zay-ia815', release='4.18.0-305.57.1.el8_4.x86_64', version='#1 SMP Tue Jul 5 17:55:55 EDT 2022', machine='x86_64')
[I 2022-10-19 17:58:37] ...ytorch/bootstrap/run.py.80: Available GPUs: [0]
[I 2022-10-19 17:58:37] ...strap/engines/factory.py.9: Creating engine...
[I 2022-10-19 17:58:37] ...trap/datasets/factory.py.8: Creating dataset...
[I 2022-10-19 17:58:51] ...m/datasets/recipe1m.py.152: Load VCs...
[I 2022-10-19 17:58:52] ...m/datasets/recipe1m.py.165: randkw_p... None
[I 2022-10-19 17:59:52] ...rap/datasets/factory.py.21: Evaluation will take place on testset (51303 items)
[I 2022-10-19 17:59:52] ...strap/models/factory.py.21: Creating model...
[I 2022-10-19 17:59:57] ...e_networks/networks.py.203: Loading CLIP Visual model...
[I 2022-10-19 18:00:00] ...e_networks/networks.py.145: cuda
[I 2022-10-19 18:00:01] ...networks/tfood_vicha.py.49: Load VCs encoder...
[I 2022-10-19 18:00:03] ...els/metrics/trijoint.py.46: self.save_ids False
[I 2022-10-19 18:00:03] ...strap/models/factory.py.44: Enabling CUDA mode...
[I 2022-10-19 18:00:09] ...etworks/tfood_vicha.py.184: training aux kw modules
[I 2022-10-19 18:00:09] ...etworks/tfood_vicha.py.184: training aux kw modules
[I 2022-10-19 18:00:09] ...otstrap/views/factory.py.8: Creating views...
[I 2022-10-19 18:00:14] ...strap/engines/engine.py.98: Loading best_eval_epoch.metric.recall_at_1_im2recipe_mean checkpoint
[I 2022-10-19 18:00:14] ...trap/engines/engine.py.440: Loading model...
[I 2022-10-19 18:00:21] ...trap/engines/engine.py.453: Loading engine...
[I 2022-10-19 18:00:21] ...trap/engines/engine.py.108: Launching evaluation procedures
[I 2022-10-19 18:00:21] ...trap/engines/engine.py.268: Evaluating model on testset for epoch 118
[I 2022-10-19 18:00:21] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler.
[I 2022-10-19 18:00:25] ...trap/engines/engine.py.321: eval: epoch 118 | batch 0/512
[I 2022-10-19 18:00:25] ...trap/engines/engine.py.325:       elapsed: 0:00:04 | left: 0:40:11
[I 2022-10-19 18:00:25] ...trap/engines/engine.py.326:       process: 4.71052 | load: 1.98279
[I 2022-10-19 18:00:38] ...trap/engines/engine.py.321: eval: epoch 118 | batch 10/512
[I 2022-10-19 18:00:38] ...trap/engines/engine.py.325:       elapsed: 0:00:17 | left: 0:13:52
[I 2022-10-19 18:00:38] ...trap/engines/engine.py.326:       process: 1.17364 | load: 0.00016
[I 2022-10-19 18:00:51] ...trap/engines/engine.py.321: eval: epoch 118 | batch 20/512
[I 2022-10-19 18:00:51] ...trap/engines/engine.py.325:       elapsed: 0:00:30 | left: 0:10:15
[I 2022-10-19 18:00:51] ...trap/engines/engine.py.326:       process: 1.16621 | load: 0.00015
[I 2022-10-19 18:01:03] ...trap/engines/engine.py.321: eval: epoch 118 | batch 30/512
[I 2022-10-19 18:01:03] ...trap/engines/engine.py.325:       elapsed: 0:00:42 | left: 0:09:49
[I 2022-10-19 18:01:03] ...trap/engines/engine.py.326:       process: 1.18304 | load: 0.00017
[I 2022-10-19 18:01:15] ...trap/engines/engine.py.321: eval: epoch 118 | batch 40/512
[I 2022-10-19 18:01:15] ...trap/engines/engine.py.325:       elapsed: 0:00:54 | left: 0:09:15
[I 2022-10-19 18:01:15] ...trap/engines/engine.py.326:       process: 1.16295 | load: 0.00018
[I 2022-10-19 18:01:27] ...trap/engines/engine.py.321: eval: epoch 118 | batch 50/512
[I 2022-10-19 18:01:27] ...trap/engines/engine.py.325:       elapsed: 0:01:06 | left: 0:09:27
[I 2022-10-19 18:01:27] ...trap/engines/engine.py.326:       process: 1.27308 | load: 0.00018
[I 2022-10-19 18:01:38] ...trap/engines/engine.py.321: eval: epoch 118 | batch 60/512
[I 2022-10-19 18:01:38] ...trap/engines/engine.py.325:       elapsed: 0:01:17 | left: 0:08:41
[I 2022-10-19 18:01:38] ...trap/engines/engine.py.326:       process: 1.13815 | load: 0.00016
[I 2022-10-19 18:01:50] ...trap/engines/engine.py.321: eval: epoch 118 | batch 70/512
[I 2022-10-19 18:01:50] ...trap/engines/engine.py.325:       elapsed: 0:01:29 | left: 0:08:36
[I 2022-10-19 18:01:50] ...trap/engines/engine.py.326:       process: 1.20964 | load: 0.00017
[I 2022-10-19 18:02:02] ...trap/engines/engine.py.321: eval: epoch 118 | batch 80/512
[I 2022-10-19 18:02:02] ...trap/engines/engine.py.325:       elapsed: 0:01:41 | left: 0:08:35
[I 2022-10-19 18:02:02] ...trap/engines/engine.py.326:       process: 1.21452 | load: 0.00016
[I 2022-10-19 18:02:14] ...trap/engines/engine.py.321: eval: epoch 118 | batch 90/512
[I 2022-10-19 18:02:14] ...trap/engines/engine.py.325:       elapsed: 0:01:53 | left: 0:08:15
[I 2022-10-19 18:02:14] ...trap/engines/engine.py.326:       process: 1.17783 | load: 0.00017
[I 2022-10-19 18:02:26] ...trap/engines/engine.py.321: eval: epoch 118 | batch 100/512
[I 2022-10-19 18:02:26] ...trap/engines/engine.py.325:       elapsed: 0:02:05 | left: 0:08:14
[I 2022-10-19 18:02:26] ...trap/engines/engine.py.326:       process: 1.26786 | load: 0.00018
[I 2022-10-19 18:02:38] ...trap/engines/engine.py.321: eval: epoch 118 | batch 110/512
[I 2022-10-19 18:02:38] ...trap/engines/engine.py.325:       elapsed: 0:02:17 | left: 0:08:03
[I 2022-10-19 18:02:38] ...trap/engines/engine.py.326:       process: 1.24228 | load: 0.00017
[I 2022-10-19 18:02:50] ...trap/engines/engine.py.321: eval: epoch 118 | batch 120/512
[I 2022-10-19 18:02:50] ...trap/engines/engine.py.325:       elapsed: 0:02:29 | left: 0:08:06
[I 2022-10-19 18:02:50] ...trap/engines/engine.py.326:       process: 1.24792 | load: 0.00017
[I 2022-10-19 18:03:02] ...trap/engines/engine.py.321: eval: epoch 118 | batch 130/512
[I 2022-10-19 18:03:02] ...trap/engines/engine.py.325:       elapsed: 0:02:41 | left: 0:07:37
[I 2022-10-19 18:03:02] ...trap/engines/engine.py.326:       process: 1.15506 | load: 0.00018
[I 2022-10-19 18:03:14] ...trap/engines/engine.py.321: eval: epoch 118 | batch 140/512
[I 2022-10-19 18:03:14] ...trap/engines/engine.py.325:       elapsed: 0:02:53 | left: 0:07:17
[I 2022-10-19 18:03:14] ...trap/engines/engine.py.326:       process: 1.17807 | load: 0.00018
[I 2022-10-19 18:03:26] ...trap/engines/engine.py.321: eval: epoch 118 | batch 150/512
[I 2022-10-19 18:03:26] ...trap/engines/engine.py.325:       elapsed: 0:03:05 | left: 0:07:12
[I 2022-10-19 18:03:26] ...trap/engines/engine.py.326:       process: 1.22876 | load: 0.00018
[I 2022-10-19 18:03:37] ...trap/engines/engine.py.321: eval: epoch 118 | batch 160/512
[I 2022-10-19 18:03:37] ...trap/engines/engine.py.325:       elapsed: 0:03:16 | left: 0:06:52
[I 2022-10-19 18:03:37] ...trap/engines/engine.py.326:       process: 1.15927 | load: 0.00017
[I 2022-10-19 18:03:49] ...trap/engines/engine.py.321: eval: epoch 118 | batch 170/512
[I 2022-10-19 18:03:49] ...trap/engines/engine.py.325:       elapsed: 0:03:28 | left: 0:06:45
[I 2022-10-19 18:03:49] ...trap/engines/engine.py.326:       process: 1.19404 | load: 0.00017
[I 2022-10-19 18:04:01] ...trap/engines/engine.py.321: eval: epoch 118 | batch 180/512
[I 2022-10-19 18:04:01] ...trap/engines/engine.py.325:       elapsed: 0:03:40 | left: 0:06:37
[I 2022-10-19 18:04:01] ...trap/engines/engine.py.326:       process: 1.24688 | load: 0.00018
[I 2022-10-19 18:04:13] ...trap/engines/engine.py.321: eval: epoch 118 | batch 190/512
[I 2022-10-19 18:04:13] ...trap/engines/engine.py.325:       elapsed: 0:03:52 | left: 0:06:15
[I 2022-10-19 18:04:13] ...trap/engines/engine.py.326:       process: 1.15304 | load: 0.00017
[I 2022-10-19 18:04:25] ...trap/engines/engine.py.321: eval: epoch 118 | batch 200/512
[I 2022-10-19 18:04:25] ...trap/engines/engine.py.325:       elapsed: 0:04:04 | left: 0:06:03
[I 2022-10-19 18:04:25] ...trap/engines/engine.py.326:       process: 1.14763 | load: 0.00017
[I 2022-10-19 18:04:36] ...trap/engines/engine.py.321: eval: epoch 118 | batch 210/512
[I 2022-10-19 18:04:36] ...trap/engines/engine.py.325:       elapsed: 0:04:15 | left: 0:05:54
[I 2022-10-19 18:04:36] ...trap/engines/engine.py.326:       process: 1.18376 | load: 0.00017
[I 2022-10-19 18:04:48] ...trap/engines/engine.py.321: eval: epoch 118 | batch 220/512
[I 2022-10-19 18:04:48] ...trap/engines/engine.py.325:       elapsed: 0:04:27 | left: 0:05:41
[I 2022-10-19 18:04:48] ...trap/engines/engine.py.326:       process: 1.17937 | load: 0.00017
[I 2022-10-19 18:05:00] ...trap/engines/engine.py.321: eval: epoch 118 | batch 230/512
[I 2022-10-19 18:05:00] ...trap/engines/engine.py.325:       elapsed: 0:04:39 | left: 0:05:31
[I 2022-10-19 18:05:00] ...trap/engines/engine.py.326:       process: 1.17158 | load: 0.00018
[I 2022-10-19 18:05:12] ...trap/engines/engine.py.321: eval: epoch 118 | batch 240/512
[I 2022-10-19 18:05:12] ...trap/engines/engine.py.325:       elapsed: 0:04:51 | left: 0:05:22
[I 2022-10-19 18:05:12] ...trap/engines/engine.py.326:       process: 1.17617 | load: 0.00017
[I 2022-10-19 18:05:23] ...trap/engines/engine.py.321: eval: epoch 118 | batch 250/512
[I 2022-10-19 18:05:23] ...trap/engines/engine.py.325:       elapsed: 0:05:02 | left: 0:05:11
[I 2022-10-19 18:05:23] ...trap/engines/engine.py.326:       process: 1.19700 | load: 0.00017
[I 2022-10-19 18:05:35] ...trap/engines/engine.py.321: eval: epoch 118 | batch 260/512
[I 2022-10-19 18:05:35] ...trap/engines/engine.py.325:       elapsed: 0:05:14 | left: 0:05:03
[I 2022-10-19 18:05:35] ...trap/engines/engine.py.326:       process: 1.20727 | load: 0.00017
[I 2022-10-19 18:05:48] ...trap/engines/engine.py.321: eval: epoch 118 | batch 270/512
[I 2022-10-19 18:05:48] ...trap/engines/engine.py.325:       elapsed: 0:05:27 | left: 0:04:52
[I 2022-10-19 18:05:48] ...trap/engines/engine.py.326:       process: 1.16677 | load: 0.00017
[I 2022-10-19 18:06:00] ...trap/engines/engine.py.321: eval: epoch 118 | batch 280/512
[I 2022-10-19 18:06:00] ...trap/engines/engine.py.325:       elapsed: 0:05:39 | left: 0:04:38
[I 2022-10-19 18:06:00] ...trap/engines/engine.py.326:       process: 1.23980 | load: 0.00018
[I 2022-10-19 18:06:11] ...trap/engines/engine.py.321: eval: epoch 118 | batch 290/512
[I 2022-10-19 18:06:11] ...trap/engines/engine.py.325:       elapsed: 0:05:50 | left: 0:04:25
[I 2022-10-19 18:06:11] ...trap/engines/engine.py.326:       process: 1.19608 | load: 0.00018
[I 2022-10-19 18:06:23] ...trap/engines/engine.py.321: eval: epoch 118 | batch 300/512
[I 2022-10-19 18:06:23] ...trap/engines/engine.py.325:       elapsed: 0:06:02 | left: 0:04:12
[I 2022-10-19 18:06:23] ...trap/engines/engine.py.326:       process: 1.19558 | load: 0.00017
[I 2022-10-19 18:06:35] ...trap/engines/engine.py.321: eval: epoch 118 | batch 310/512
[I 2022-10-19 18:06:35] ...trap/engines/engine.py.325:       elapsed: 0:06:14 | left: 0:04:00
[I 2022-10-19 18:06:35] ...trap/engines/engine.py.326:       process: 1.17222 | load: 0.00019
[I 2022-10-19 18:06:47] ...trap/engines/engine.py.321: eval: epoch 118 | batch 320/512
[I 2022-10-19 18:06:47] ...trap/engines/engine.py.325:       elapsed: 0:06:26 | left: 0:03:44
[I 2022-10-19 18:06:47] ...trap/engines/engine.py.326:       process: 1.15607 | load: 0.00018
[I 2022-10-19 18:06:59] ...trap/engines/engine.py.321: eval: epoch 118 | batch 330/512
[I 2022-10-19 18:06:59] ...trap/engines/engine.py.325:       elapsed: 0:06:38 | left: 0:03:35
[I 2022-10-19 18:06:59] ...trap/engines/engine.py.326:       process: 1.19169 | load: 0.00019
[I 2022-10-19 18:07:11] ...trap/engines/engine.py.321: eval: epoch 118 | batch 340/512
[I 2022-10-19 18:07:11] ...trap/engines/engine.py.325:       elapsed: 0:06:50 | left: 0:03:24
[I 2022-10-19 18:07:11] ...trap/engines/engine.py.326:       process: 1.19696 | load: 0.00019
[I 2022-10-19 18:07:23] ...trap/engines/engine.py.321: eval: epoch 118 | batch 350/512
[I 2022-10-19 18:07:23] ...trap/engines/engine.py.325:       elapsed: 0:07:02 | left: 0:03:11
[I 2022-10-19 18:07:23] ...trap/engines/engine.py.326:       process: 1.17722 | load: 0.00019
[I 2022-10-19 18:07:35] ...trap/engines/engine.py.321: eval: epoch 118 | batch 360/512
[I 2022-10-19 18:07:35] ...trap/engines/engine.py.325:       elapsed: 0:07:14 | left: 0:03:04
[I 2022-10-19 18:07:35] ...trap/engines/engine.py.326:       process: 1.22421 | load: 0.00019
[I 2022-10-19 18:07:47] ...trap/engines/engine.py.321: eval: epoch 118 | batch 370/512
[I 2022-10-19 18:07:47] ...trap/engines/engine.py.325:       elapsed: 0:07:26 | left: 0:02:53
[I 2022-10-19 18:07:47] ...trap/engines/engine.py.326:       process: 1.23092 | load: 0.00018
[I 2022-10-19 18:07:59] ...trap/engines/engine.py.321: eval: epoch 118 | batch 380/512
[I 2022-10-19 18:07:59] ...trap/engines/engine.py.325:       elapsed: 0:07:38 | left: 0:02:38
[I 2022-10-19 18:07:59] ...trap/engines/engine.py.326:       process: 1.20303 | load: 0.00018
[I 2022-10-19 18:08:11] ...trap/engines/engine.py.321: eval: epoch 118 | batch 390/512
[I 2022-10-19 18:08:11] ...trap/engines/engine.py.325:       elapsed: 0:07:50 | left: 0:02:26
[I 2022-10-19 18:08:11] ...trap/engines/engine.py.326:       process: 1.20233 | load: 0.00018
[I 2022-10-19 18:08:23] ...trap/engines/engine.py.321: eval: epoch 118 | batch 400/512
[I 2022-10-19 18:08:23] ...trap/engines/engine.py.325:       elapsed: 0:08:02 | left: 0:02:09
[I 2022-10-19 18:08:23] ...trap/engines/engine.py.326:       process: 1.06510 | load: 0.00018
[I 2022-10-19 18:08:34] ...trap/engines/engine.py.321: eval: epoch 118 | batch 410/512
[I 2022-10-19 18:08:34] ...trap/engines/engine.py.325:       elapsed: 0:08:13 | left: 0:01:52
[I 2022-10-19 18:08:34] ...trap/engines/engine.py.326:       process: 1.09548 | load: 0.00018
[I 2022-10-19 18:08:45] ...trap/engines/engine.py.321: eval: epoch 118 | batch 420/512
[I 2022-10-19 18:08:45] ...trap/engines/engine.py.325:       elapsed: 0:08:24 | left: 0:01:40
[I 2022-10-19 18:08:45] ...trap/engines/engine.py.326:       process: 1.10816 | load: 0.00018
[I 2022-10-19 18:08:55] ...trap/engines/engine.py.321: eval: epoch 118 | batch 430/512
[I 2022-10-19 18:08:55] ...trap/engines/engine.py.325:       elapsed: 0:08:34 | left: 0:01:28
[I 2022-10-19 18:08:55] ...trap/engines/engine.py.326:       process: 1.10086 | load: 0.00017
[I 2022-10-19 18:09:06] ...trap/engines/engine.py.321: eval: epoch 118 | batch 440/512
[I 2022-10-19 18:09:06] ...trap/engines/engine.py.325:       elapsed: 0:08:45 | left: 0:01:19
[I 2022-10-19 18:09:06] ...trap/engines/engine.py.326:       process: 1.11059 | load: 0.00021
[I 2022-10-19 18:09:17] ...trap/engines/engine.py.321: eval: epoch 118 | batch 450/512
[I 2022-10-19 18:09:17] ...trap/engines/engine.py.325:       elapsed: 0:08:56 | left: 0:01:08
[I 2022-10-19 18:09:17] ...trap/engines/engine.py.326:       process: 1.09968 | load: 0.00019
[I 2022-10-19 18:09:28] ...trap/engines/engine.py.321: eval: epoch 118 | batch 460/512
[I 2022-10-19 18:09:28] ...trap/engines/engine.py.325:       elapsed: 0:09:07 | left: 0:00:57
[I 2022-10-19 18:09:28] ...trap/engines/engine.py.326:       process: 1.09168 | load: 0.00019
[I 2022-10-19 18:09:39] ...trap/engines/engine.py.321: eval: epoch 118 | batch 470/512
[I 2022-10-19 18:09:39] ...trap/engines/engine.py.325:       elapsed: 0:09:18 | left: 0:00:45
[I 2022-10-19 18:09:39] ...trap/engines/engine.py.326:       process: 1.07047 | load: 0.00019
[I 2022-10-19 18:09:50] ...trap/engines/engine.py.321: eval: epoch 118 | batch 480/512
[I 2022-10-19 18:09:50] ...trap/engines/engine.py.325:       elapsed: 0:09:29 | left: 0:00:35
[I 2022-10-19 18:09:50] ...trap/engines/engine.py.326:       process: 1.09794 | load: 0.00018
[I 2022-10-19 18:10:01] ...trap/engines/engine.py.321: eval: epoch 118 | batch 490/512
[I 2022-10-19 18:10:01] ...trap/engines/engine.py.325:       elapsed: 0:09:40 | left: 0:00:24
[I 2022-10-19 18:10:01] ...trap/engines/engine.py.326:       process: 1.10590 | load: 0.00019
[I 2022-10-19 18:10:12] ...trap/engines/engine.py.321: eval: epoch 118 | batch 500/512
[I 2022-10-19 18:10:12] ...trap/engines/engine.py.325:       elapsed: 0:09:51 | left: 0:00:13
[I 2022-10-19 18:10:12] ...trap/engines/engine.py.326:       process: 1.10654 | load: 0.00018
[I 2022-10-19 18:10:23] ...trap/engines/engine.py.321: eval: epoch 118 | batch 510/512
[I 2022-10-19 18:10:23] ...trap/engines/engine.py.325:       elapsed: 0:10:02 | left: 0:00:02
[I 2022-10-19 18:10:23] ...trap/engines/engine.py.326:       process: 1.11018 | load: 0.00016
[S 2022-10-19 18:10:26] ...trap/engines/engine.py.113: eval_epoch.epoch: 118
[I 2022-10-19 18:10:26] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 1 x 40000 matchs
[S 2022-10-19 18:18:37] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 4.00
[S 2022-10-19 18:18:37] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 4.00
[S 2022-10-19 18:18:37] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.317925
[S 2022-10-19 18:18:37] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.563475
[S 2022-10-19 18:18:37] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.664475
[S 2022-10-19 18:18:37] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.304550
[S 2022-10-19 18:18:37] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.552050
[S 2022-10-19 18:18:37] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.653925
[S 2022-10-19 18:18:37] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000
[S 2022-10-19 18:18:37] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000
[S 2022-10-19 18:18:37] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.000000
[S 2022-10-19 18:18:37] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.000000
[S 2022-10-19 18:18:37] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.000000
[S 2022-10-19 18:18:37] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.000000
[S 2022-10-19 18:18:37] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.000000
[S 2022-10-19 18:18:37] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.000000
[I 2022-10-19 18:19:39] ...trap/engines/engine.py.115: Ending evaluation procedures
[W 2022-10-19 18:19:43] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json'
[W 2022-10-19 18:19:43] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[W 2022-10-19 18:19:43] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[I 2022-10-19 18:19:52] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html
[I 2022-10-19 20:28:15] ...ytorch/bootstrap/run.py.73: Saving environment info
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75: options: 
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:   dataset:
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     aux_kw_path:
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       test: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_test_titles_kw.json
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       train: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_train_titles_kw.json
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       val: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_val_titles_kw.json
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     batch_sampler: triplet_classif
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     batch_size: 100
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     database: lmdb
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     dir: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     eval_split: test
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     freq_mismatch: 0.0
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     image_from: database
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     import: recipe1m.datasets.factory
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     kw_path:
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       test: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_test_ingr_kw.json
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       train: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_train_ingr_kw.json
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       val: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/clip_da/layer1_val_ingr_kw.json
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     name: recipe1m
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     nb_classes: 1048
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     nb_threads: 4
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     randkw_p: 0.3
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     randkw_p_aux: 0.5
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     tokenized_raw_text: True
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     train_split: None
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     use_vcs: True
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:   engine:
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     debug: False
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     name: logger
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     nb_epochs: 120
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     print_freq: 10
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     saving_criteria: ['eval_epoch.metric.med_im2recipe_mean:min', 'eval_epoch.metric.recall_at_1_im2recipe_mean:max']
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:   exp:
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     checkpoint: None
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     dir: /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     resume: best_eval_epoch.metric.recall_at_1_im2recipe_mean
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:   misc:
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     cuda: True
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     data_parrallel: False
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     logs_name: None
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     overrite: False
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     seed: 1338
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:   model:
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     criterion:
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:         tri_cross_encoder: False
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       itm_loss_weight: 1
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       keep_background: False
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       retrieval_strategy:
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:         aggregation: valid
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:         get_tokens: True
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:         margin: 0.05
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:         margin_params:
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:           increment: 0.005
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:           increment_margin: True
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:           max_margin: 0.3
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:         name: triplet
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:         nb_samples: 9999
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:         sampling: max_negative
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:         substrategy: ['IRR', 'RII', 'SIRR', 'SRII']
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:         substrategy_weights: [1.0, 1.0, 0.1, 0.1]
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     import: recipe1m.models.factory
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     metric:
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       k_test: 10
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       nb_bags: 1
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       nb_matchs_per_bag: 50000
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       retrieval_dir: /gpfsscratch/rech/dyf/ugz83ue/tmp/recipe1m
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       trijoint: True
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     name: trijoint
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     network:
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       activations: ['tanh', 'normalize']
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       aux_kwords: True
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       aux_kwords_encoder: True
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       avg_concat: True
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       bert_config: /linkhome/rech/genisi01/ugz83ue/tfood/recipe1m/models/networks/recipe_networks/config_bert_albef.json
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       cat_pos: 0
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       checkpoint: None
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       class_attention: False
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       class_attention_fus: False
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       cls_token: False
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       cls_token_fus: False
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       cross_decoder_image: True
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       cross_decoder_recipe: True
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       cross_encoder: True
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       cross_encoder_params:
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:         class_attention: False
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:         cls_token: False
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:         cross_attention: False
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:         cross_decoder: True
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:         double_decoder: False
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:         get_tokens: False
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:         load_backbones: False
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:         n_heads: 4
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:         n_layers: 4
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:         query_img: False
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       cross_transformer: False
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       dim_emb: 1024
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       dim_image_out: 2048
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       dim_ingr_out: 300
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       dim_instr_in: 1024
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       dim_instr_out: 1024
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       embedding_names: ['proj_image']
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       freeze_im: False
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       freeze_rec: False
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       get_tokens: True
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       get_tokens_cross_decoder_recipe: True
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       hidden_size: 512
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       image_backbone_name: clip
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       image_res: 224
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       kwords_same_level: False
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       last_cat_aux: True
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       n_heads: 4
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       n_heads_cross: 4
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       n_heads_cross_image: 4
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       n_heads_single: 4
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       n_layers: 2
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       n_layers_cross: 2
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       n_layers_cross_image: 1
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       n_layers_single: 2
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       name: trijoint
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       num_hidden_layers_kw: 2
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       old: True
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       path_ingrs: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/text/vocab.pkl
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       path_vocab: /gpfsscratch/rech/dyf/ugz83ue/data/recipe1m/text/vocab_all.txt
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       recipe_encoder: tfoodvicha-transformer
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       text_encoder: bert-base-uncased
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       vision_width: 768
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       vit_all_tokens: True
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       with_ingrs: True
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       with_instrs: True
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:       with_titles: True
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     with_classif: False
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:   optimizer:
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     batch_size_factor: None
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     clip_grad: 8.0
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     import: recipe1m.optimizers.factory
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     lr: 1e-05
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     lr_img: 1e-06
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     name: trijoint_fixed_fine_tune
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:     switch_epoch: 20
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:   path_opts: /linkhome/rech/genisi01/ugz83ue/tfood/recipe1m/options/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_noimgproj_keepall.yaml
[S 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.75:   view: ['logs:train_epoch.loss', 'logs:train_epoch.bad_pairs', 'logs:eval_epoch.metric.med_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_1_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_5_im2recipe_mean', 'logs:eval_epoch.metric.recall_at_10_im2recipe_mean', 'logs:eval_epoch.metric.med_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_1_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_5_recipe2im_mean', 'logs:eval_epoch.metric.recall_at_10_recipe2im_mean', 'logs:optimizer.is_optimizer_recipe&image', 'logs:optimizer.total_norm']
[I 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.76: posix.uname_result(sysname='Linux', nodename='jean-zay-ia819', release='4.18.0-305.57.1.el8_4.x86_64', version='#1 SMP Tue Jul 5 17:55:55 EDT 2022', machine='x86_64')
[I 2022-10-19 20:28:22] ...ytorch/bootstrap/run.py.80: Available GPUs: [0]
[I 2022-10-19 20:28:22] ...strap/engines/factory.py.9: Creating engine...
[I 2022-10-19 20:28:22] ...trap/datasets/factory.py.8: Creating dataset...
[I 2022-10-19 20:28:37] ...m/datasets/recipe1m.py.152: Load VCs...
[I 2022-10-19 20:28:37] ...m/datasets/recipe1m.py.165: randkw_p... None
[I 2022-10-19 20:29:38] ...rap/datasets/factory.py.21: Evaluation will take place on testset (51303 items)
[I 2022-10-19 20:29:38] ...strap/models/factory.py.21: Creating model...
[I 2022-10-19 20:29:42] ...e_networks/networks.py.203: Loading CLIP Visual model...
[I 2022-10-19 20:29:46] ...e_networks/networks.py.145: cuda
[I 2022-10-19 20:29:47] ...networks/tfood_vicha.py.49: Load VCs encoder...
[I 2022-10-19 20:29:48] ...els/metrics/trijoint.py.46: self.save_ids False
[I 2022-10-19 20:29:48] ...strap/models/factory.py.44: Enabling CUDA mode...
[I 2022-10-19 20:29:55] ...etworks/tfood_vicha.py.184: training aux kw modules
[I 2022-10-19 20:29:55] ...etworks/tfood_vicha.py.184: training aux kw modules
[I 2022-10-19 20:29:55] ...otstrap/views/factory.py.8: Creating views...
[I 2022-10-19 20:29:59] ...strap/engines/engine.py.98: Loading best_eval_epoch.metric.recall_at_1_im2recipe_mean checkpoint
[I 2022-10-19 20:29:59] ...trap/engines/engine.py.440: Loading model...
[I 2022-10-19 20:30:06] ...trap/engines/engine.py.453: Loading engine...
[I 2022-10-19 20:30:06] ...trap/engines/engine.py.108: Launching evaluation procedures
[I 2022-10-19 20:30:06] ...trap/engines/engine.py.268: Evaluating model on testset for epoch 118
[I 2022-10-19 20:30:06] ...m/datasets/recipe1m.py.384: Dataset will be sampled with "random" batch_sampler.
[I 2022-10-19 20:30:10] ...trap/engines/engine.py.321: eval: epoch 118 | batch 0/512
[I 2022-10-19 20:30:10] ...trap/engines/engine.py.325:       elapsed: 0:00:04 | left: 0:39:22
[I 2022-10-19 20:30:10] ...trap/engines/engine.py.326:       process: 4.61330 | load: 2.05310
[I 2022-10-19 20:30:23] ...trap/engines/engine.py.321: eval: epoch 118 | batch 10/512
[I 2022-10-19 20:30:23] ...trap/engines/engine.py.325:       elapsed: 0:00:17 | left: 0:13:40
[I 2022-10-19 20:30:23] ...trap/engines/engine.py.326:       process: 1.30924 | load: 0.00016
[I 2022-10-19 20:30:35] ...trap/engines/engine.py.321: eval: epoch 118 | batch 20/512
[I 2022-10-19 20:30:35] ...trap/engines/engine.py.325:       elapsed: 0:00:29 | left: 0:10:26
[I 2022-10-19 20:30:35] ...trap/engines/engine.py.326:       process: 1.31537 | load: 0.00017
[I 2022-10-19 20:30:47] ...trap/engines/engine.py.321: eval: epoch 118 | batch 30/512
[I 2022-10-19 20:30:47] ...trap/engines/engine.py.325:       elapsed: 0:00:41 | left: 0:09:43
[I 2022-10-19 20:30:47] ...trap/engines/engine.py.326:       process: 1.21168 | load: 0.00017
[I 2022-10-19 20:30:59] ...trap/engines/engine.py.321: eval: epoch 118 | batch 40/512
[I 2022-10-19 20:30:59] ...trap/engines/engine.py.325:       elapsed: 0:00:53 | left: 0:09:35
[I 2022-10-19 20:30:59] ...trap/engines/engine.py.326:       process: 1.17606 | load: 0.00016
[I 2022-10-19 20:31:12] ...trap/engines/engine.py.321: eval: epoch 118 | batch 50/512
[I 2022-10-19 20:31:12] ...trap/engines/engine.py.325:       elapsed: 0:01:05 | left: 0:09:30
[I 2022-10-19 20:31:12] ...trap/engines/engine.py.326:       process: 1.33627 | load: 0.00017
[I 2022-10-19 20:31:24] ...trap/engines/engine.py.321: eval: epoch 118 | batch 60/512
[I 2022-10-19 20:31:24] ...trap/engines/engine.py.325:       elapsed: 0:01:18 | left: 0:09:30
[I 2022-10-19 20:31:24] ...trap/engines/engine.py.326:       process: 1.32849 | load: 0.00016
[I 2022-10-19 20:31:37] ...trap/engines/engine.py.321: eval: epoch 118 | batch 70/512
[I 2022-10-19 20:31:37] ...trap/engines/engine.py.325:       elapsed: 0:01:30 | left: 0:09:15
[I 2022-10-19 20:31:37] ...trap/engines/engine.py.326:       process: 1.24423 | load: 0.00017
[I 2022-10-19 20:31:49] ...trap/engines/engine.py.321: eval: epoch 118 | batch 80/512
[I 2022-10-19 20:31:49] ...trap/engines/engine.py.325:       elapsed: 0:01:43 | left: 0:08:55
[I 2022-10-19 20:31:49] ...trap/engines/engine.py.326:       process: 1.27199 | load: 0.00017
[I 2022-10-19 20:32:01] ...trap/engines/engine.py.321: eval: epoch 118 | batch 90/512
[I 2022-10-19 20:32:01] ...trap/engines/engine.py.325:       elapsed: 0:01:55 | left: 0:08:39
[I 2022-10-19 20:32:01] ...trap/engines/engine.py.326:       process: 1.24922 | load: 0.00018
[I 2022-10-19 20:32:13] ...trap/engines/engine.py.321: eval: epoch 118 | batch 100/512
[I 2022-10-19 20:32:13] ...trap/engines/engine.py.325:       elapsed: 0:02:07 | left: 0:08:15
[I 2022-10-19 20:32:13] ...trap/engines/engine.py.326:       process: 1.22047 | load: 0.00018
[I 2022-10-19 20:32:25] ...trap/engines/engine.py.321: eval: epoch 118 | batch 110/512
[I 2022-10-19 20:32:25] ...trap/engines/engine.py.325:       elapsed: 0:02:19 | left: 0:08:05
[I 2022-10-19 20:32:25] ...trap/engines/engine.py.326:       process: 1.21153 | load: 0.00016
[I 2022-10-19 20:32:38] ...trap/engines/engine.py.321: eval: epoch 118 | batch 120/512
[I 2022-10-19 20:32:38] ...trap/engines/engine.py.325:       elapsed: 0:02:31 | left: 0:08:03
[I 2022-10-19 20:32:38] ...trap/engines/engine.py.326:       process: 1.22327 | load: 0.00017
[I 2022-10-19 20:32:50] ...trap/engines/engine.py.321: eval: epoch 118 | batch 130/512
[I 2022-10-19 20:32:50] ...trap/engines/engine.py.325:       elapsed: 0:02:44 | left: 0:07:50
[I 2022-10-19 20:32:50] ...trap/engines/engine.py.326:       process: 1.28036 | load: 0.00018
[I 2022-10-19 20:33:02] ...trap/engines/engine.py.321: eval: epoch 118 | batch 140/512
[I 2022-10-19 20:33:02] ...trap/engines/engine.py.325:       elapsed: 0:02:56 | left: 0:07:39
[I 2022-10-19 20:33:02] ...trap/engines/engine.py.326:       process: 1.31209 | load: 0.00017
[I 2022-10-19 20:33:15] ...trap/engines/engine.py.321: eval: epoch 118 | batch 150/512
[I 2022-10-19 20:33:15] ...trap/engines/engine.py.325:       elapsed: 0:03:08 | left: 0:07:33
[I 2022-10-19 20:33:15] ...trap/engines/engine.py.326:       process: 1.24347 | load: 0.00017
[I 2022-10-19 20:33:26] ...trap/engines/engine.py.321: eval: epoch 118 | batch 160/512
[I 2022-10-19 20:33:26] ...trap/engines/engine.py.325:       elapsed: 0:03:20 | left: 0:07:03
[I 2022-10-19 20:33:26] ...trap/engines/engine.py.326:       process: 1.21875 | load: 0.00017
[I 2022-10-19 20:33:38] ...trap/engines/engine.py.321: eval: epoch 118 | batch 170/512
[I 2022-10-19 20:33:38] ...trap/engines/engine.py.325:       elapsed: 0:03:32 | left: 0:06:51
[I 2022-10-19 20:33:38] ...trap/engines/engine.py.326:       process: 1.19466 | load: 0.00017
[I 2022-10-19 20:33:50] ...trap/engines/engine.py.321: eval: epoch 118 | batch 180/512
[I 2022-10-19 20:33:50] ...trap/engines/engine.py.325:       elapsed: 0:03:44 | left: 0:06:38
[I 2022-10-19 20:33:50] ...trap/engines/engine.py.326:       process: 1.18740 | load: 0.00018
[I 2022-10-19 20:34:02] ...trap/engines/engine.py.321: eval: epoch 118 | batch 190/512
[I 2022-10-19 20:34:02] ...trap/engines/engine.py.325:       elapsed: 0:03:56 | left: 0:06:26
[I 2022-10-19 20:34:02] ...trap/engines/engine.py.326:       process: 1.16282 | load: 0.00017
[I 2022-10-19 20:34:14] ...trap/engines/engine.py.321: eval: epoch 118 | batch 200/512
[I 2022-10-19 20:34:14] ...trap/engines/engine.py.325:       elapsed: 0:04:08 | left: 0:06:09
[I 2022-10-19 20:34:14] ...trap/engines/engine.py.326:       process: 1.15835 | load: 0.00017
[I 2022-10-19 20:34:26] ...trap/engines/engine.py.321: eval: epoch 118 | batch 210/512
[I 2022-10-19 20:34:26] ...trap/engines/engine.py.325:       elapsed: 0:04:20 | left: 0:05:58
[I 2022-10-19 20:34:26] ...trap/engines/engine.py.326:       process: 1.17714 | load: 0.00017
[I 2022-10-19 20:34:38] ...trap/engines/engine.py.321: eval: epoch 118 | batch 220/512
[I 2022-10-19 20:34:38] ...trap/engines/engine.py.325:       elapsed: 0:04:32 | left: 0:05:46
[I 2022-10-19 20:34:38] ...trap/engines/engine.py.326:       process: 1.19533 | load: 0.00017
[I 2022-10-19 20:34:50] ...trap/engines/engine.py.321: eval: epoch 118 | batch 230/512
[I 2022-10-19 20:34:50] ...trap/engines/engine.py.325:       elapsed: 0:04:44 | left: 0:05:33
[I 2022-10-19 20:34:50] ...trap/engines/engine.py.326:       process: 1.17570 | load: 0.00017
[I 2022-10-19 20:35:02] ...trap/engines/engine.py.321: eval: epoch 118 | batch 240/512
[I 2022-10-19 20:35:02] ...trap/engines/engine.py.325:       elapsed: 0:04:55 | left: 0:05:21
[I 2022-10-19 20:35:02] ...trap/engines/engine.py.326:       process: 1.17442 | load: 0.00017
[I 2022-10-19 20:35:14] ...trap/engines/engine.py.321: eval: epoch 118 | batch 250/512
[I 2022-10-19 20:35:14] ...trap/engines/engine.py.325:       elapsed: 0:05:07 | left: 0:05:13
[I 2022-10-19 20:35:14] ...trap/engines/engine.py.326:       process: 1.20782 | load: 0.00017
[I 2022-10-19 20:35:26] ...trap/engines/engine.py.321: eval: epoch 118 | batch 260/512
[I 2022-10-19 20:35:26] ...trap/engines/engine.py.325:       elapsed: 0:05:19 | left: 0:05:00
[I 2022-10-19 20:35:26] ...trap/engines/engine.py.326:       process: 1.18836 | load: 0.00018
[I 2022-10-19 20:35:37] ...trap/engines/engine.py.321: eval: epoch 118 | batch 270/512
[I 2022-10-19 20:35:37] ...trap/engines/engine.py.325:       elapsed: 0:05:31 | left: 0:04:46
[I 2022-10-19 20:35:37] ...trap/engines/engine.py.326:       process: 1.20199 | load: 0.00018
[I 2022-10-19 20:35:49] ...trap/engines/engine.py.321: eval: epoch 118 | batch 280/512
[I 2022-10-19 20:35:49] ...trap/engines/engine.py.325:       elapsed: 0:05:43 | left: 0:04:38
[I 2022-10-19 20:35:49] ...trap/engines/engine.py.326:       process: 1.20784 | load: 0.00017
[I 2022-10-19 20:36:01] ...trap/engines/engine.py.321: eval: epoch 118 | batch 290/512
[I 2022-10-19 20:36:01] ...trap/engines/engine.py.325:       elapsed: 0:05:55 | left: 0:04:26
[I 2022-10-19 20:36:01] ...trap/engines/engine.py.326:       process: 1.22435 | load: 0.00017
[I 2022-10-19 20:36:13] ...trap/engines/engine.py.321: eval: epoch 118 | batch 300/512
[I 2022-10-19 20:36:13] ...trap/engines/engine.py.325:       elapsed: 0:06:07 | left: 0:04:12
[I 2022-10-19 20:36:13] ...trap/engines/engine.py.326:       process: 1.19738 | load: 0.00019
[I 2022-10-19 20:36:25] ...trap/engines/engine.py.321: eval: epoch 118 | batch 310/512
[I 2022-10-19 20:36:25] ...trap/engines/engine.py.325:       elapsed: 0:06:19 | left: 0:04:00
[I 2022-10-19 20:36:25] ...trap/engines/engine.py.326:       process: 1.19699 | load: 0.00018
[I 2022-10-19 20:36:37] ...trap/engines/engine.py.321: eval: epoch 118 | batch 320/512
[I 2022-10-19 20:36:37] ...trap/engines/engine.py.325:       elapsed: 0:06:31 | left: 0:03:47
[I 2022-10-19 20:36:37] ...trap/engines/engine.py.326:       process: 1.16244 | load: 0.00018
[I 2022-10-19 20:36:49] ...trap/engines/engine.py.321: eval: epoch 118 | batch 330/512
[I 2022-10-19 20:36:49] ...trap/engines/engine.py.325:       elapsed: 0:06:43 | left: 0:03:39
[I 2022-10-19 20:36:49] ...trap/engines/engine.py.326:       process: 1.21502 | load: 0.00017
[I 2022-10-19 20:37:01] ...trap/engines/engine.py.321: eval: epoch 118 | batch 340/512
[I 2022-10-19 20:37:01] ...trap/engines/engine.py.325:       elapsed: 0:06:55 | left: 0:03:26
[I 2022-10-19 20:37:01] ...trap/engines/engine.py.326:       process: 1.18121 | load: 0.00020
[I 2022-10-19 20:37:13] ...trap/engines/engine.py.321: eval: epoch 118 | batch 350/512
[I 2022-10-19 20:37:13] ...trap/engines/engine.py.325:       elapsed: 0:07:07 | left: 0:03:13
[I 2022-10-19 20:37:13] ...trap/engines/engine.py.326:       process: 1.19736 | load: 0.00018
[I 2022-10-19 20:37:25] ...trap/engines/engine.py.321: eval: epoch 118 | batch 360/512
[I 2022-10-19 20:37:25] ...trap/engines/engine.py.325:       elapsed: 0:07:19 | left: 0:03:02
[I 2022-10-19 20:37:25] ...trap/engines/engine.py.326:       process: 1.19329 | load: 0.00017
[I 2022-10-19 20:37:37] ...trap/engines/engine.py.321: eval: epoch 118 | batch 370/512
[I 2022-10-19 20:37:37] ...trap/engines/engine.py.325:       elapsed: 0:07:31 | left: 0:02:50
[I 2022-10-19 20:37:37] ...trap/engines/engine.py.326:       process: 1.20107 | load: 0.00019
[I 2022-10-19 20:37:49] ...trap/engines/engine.py.321: eval: epoch 118 | batch 380/512
[I 2022-10-19 20:37:49] ...trap/engines/engine.py.325:       elapsed: 0:07:43 | left: 0:02:39
[I 2022-10-19 20:37:49] ...trap/engines/engine.py.326:       process: 1.22267 | load: 0.00017
[I 2022-10-19 20:38:01] ...trap/engines/engine.py.321: eval: epoch 118 | batch 390/512
[I 2022-10-19 20:38:01] ...trap/engines/engine.py.325:       elapsed: 0:07:55 | left: 0:02:27
[I 2022-10-19 20:38:01] ...trap/engines/engine.py.326:       process: 1.21711 | load: 0.00018
[I 2022-10-19 20:38:13] ...trap/engines/engine.py.321: eval: epoch 118 | batch 400/512
[I 2022-10-19 20:38:13] ...trap/engines/engine.py.325:       elapsed: 0:08:07 | left: 0:02:14
[I 2022-10-19 20:38:13] ...trap/engines/engine.py.326:       process: 1.18023 | load: 0.00018
[I 2022-10-19 20:38:25] ...trap/engines/engine.py.321: eval: epoch 118 | batch 410/512
[I 2022-10-19 20:38:25] ...trap/engines/engine.py.325:       elapsed: 0:08:19 | left: 0:02:04
[I 2022-10-19 20:38:25] ...trap/engines/engine.py.326:       process: 1.21780 | load: 0.00018
[I 2022-10-19 20:38:38] ...trap/engines/engine.py.321: eval: epoch 118 | batch 420/512
[I 2022-10-19 20:38:38] ...trap/engines/engine.py.325:       elapsed: 0:08:31 | left: 0:01:52
[I 2022-10-19 20:38:38] ...trap/engines/engine.py.326:       process: 1.23212 | load: 0.00019
[I 2022-10-19 20:38:50] ...trap/engines/engine.py.321: eval: epoch 118 | batch 430/512
[I 2022-10-19 20:38:50] ...trap/engines/engine.py.325:       elapsed: 0:08:43 | left: 0:01:40
[I 2022-10-19 20:38:50] ...trap/engines/engine.py.326:       process: 1.23155 | load: 0.00018
[I 2022-10-19 20:39:02] ...trap/engines/engine.py.321: eval: epoch 118 | batch 440/512
[I 2022-10-19 20:39:02] ...trap/engines/engine.py.325:       elapsed: 0:08:56 | left: 0:01:27
[I 2022-10-19 20:39:02] ...trap/engines/engine.py.326:       process: 1.22783 | load: 0.00019
[I 2022-10-19 20:39:14] ...trap/engines/engine.py.321: eval: epoch 118 | batch 450/512
[I 2022-10-19 20:39:14] ...trap/engines/engine.py.325:       elapsed: 0:09:08 | left: 0:01:15
[I 2022-10-19 20:39:14] ...trap/engines/engine.py.326:       process: 1.21678 | load: 0.00017
[I 2022-10-19 20:39:26] ...trap/engines/engine.py.321: eval: epoch 118 | batch 460/512
[I 2022-10-19 20:39:26] ...trap/engines/engine.py.325:       elapsed: 0:09:20 | left: 0:01:03
[I 2022-10-19 20:39:26] ...trap/engines/engine.py.326:       process: 1.20145 | load: 0.00018
[I 2022-10-19 20:39:38] ...trap/engines/engine.py.321: eval: epoch 118 | batch 470/512
[I 2022-10-19 20:39:38] ...trap/engines/engine.py.325:       elapsed: 0:09:32 | left: 0:00:50
[I 2022-10-19 20:39:38] ...trap/engines/engine.py.326:       process: 1.17453 | load: 0.00017
[I 2022-10-19 20:39:51] ...trap/engines/engine.py.321: eval: epoch 118 | batch 480/512
[I 2022-10-19 20:39:51] ...trap/engines/engine.py.325:       elapsed: 0:09:44 | left: 0:00:39
[I 2022-10-19 20:39:51] ...trap/engines/engine.py.326:       process: 1.21322 | load: 0.00018
[I 2022-10-19 20:40:03] ...trap/engines/engine.py.321: eval: epoch 118 | batch 490/512
[I 2022-10-19 20:40:03] ...trap/engines/engine.py.325:       elapsed: 0:09:56 | left: 0:00:26
[I 2022-10-19 20:40:03] ...trap/engines/engine.py.326:       process: 1.21616 | load: 0.00024
[I 2022-10-19 20:40:15] ...trap/engines/engine.py.321: eval: epoch 118 | batch 500/512
[I 2022-10-19 20:40:15] ...trap/engines/engine.py.325:       elapsed: 0:10:08 | left: 0:00:14
[I 2022-10-19 20:40:15] ...trap/engines/engine.py.326:       process: 1.11724 | load: 0.00019
[I 2022-10-19 20:40:26] ...trap/engines/engine.py.321: eval: epoch 118 | batch 510/512
[I 2022-10-19 20:40:26] ...trap/engines/engine.py.325:       elapsed: 0:10:19 | left: 0:00:02
[I 2022-10-19 20:40:26] ...trap/engines/engine.py.326:       process: 1.11554 | load: 0.00014
[S 2022-10-19 20:40:28] ...trap/engines/engine.py.113: eval_epoch.epoch: 118
[I 2022-10-19 20:40:28] ...els/metrics/trijoint.py.98: Computing retrieval ranking for 1 x 50000 matchs
[S 2022-10-19 20:52:30] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_mean: 5.00
[S 2022-10-19 20:52:30] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_mean: 5.00
[S 2022-10-19 20:52:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_mean: 0.293680
[S 2022-10-19 20:52:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_mean: 0.531180
[S 2022-10-19 20:52:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_mean: 0.634700
[S 2022-10-19 20:52:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_mean: 0.279280
[S 2022-10-19 20:52:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_mean: 0.518540
[S 2022-10-19 20:52:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_mean: 0.620600
[S 2022-10-19 20:52:30] ...strap/engines/engine.py.72: eval_epoch.metric.med_im2recipe_std: 0.000000
[S 2022-10-19 20:52:30] ...strap/engines/engine.py.72: eval_epoch.metric.med_recipe2im_std: 0.000000
[S 2022-10-19 20:52:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_im2recipe_std: 0.000000
[S 2022-10-19 20:52:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_im2recipe_std: 0.000000
[S 2022-10-19 20:52:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_im2recipe_std: 0.000000
[S 2022-10-19 20:52:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_1_recipe2im_std: 0.000000
[S 2022-10-19 20:52:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_5_recipe2im_std: 0.000000
[S 2022-10-19 20:52:30] ...strap/engines/engine.py.72: eval_epoch.metric.recall_at_10_recipe2im_std: 0.000000
[I 2022-10-19 20:53:46] ...trap/engines/engine.py.115: Ending evaluation procedures
[W 2022-10-19 20:53:50] ...otstrap/views/plotly.py.87: View 'train_epoch.bad_pairs' not in 'logs.json'
[W 2022-10-19 20:53:50] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[W 2022-10-19 20:53:50] ...otstrap/views/plotly.py.93: Split 'optimizer' not in colors '['train_epoch', 'train_batch', 'val_epoch', 'val_batch', 'eval_epoch', 'eval_batch', 'test_epoch', 'test_batch', 'eval_pruned_epoch', 'eval_pruned_batch']'
[I 2022-10-19 20:53:59] ...tstrap/views/plotly.py.125: Plotly view generated in /gpfswork/rech/dyf/ugz83ue/logs/recipe1m/tfood_tfoodvicha_ttl_ingr_lastcataux_auxkwenc_clipvit_vlp_init_onlytext_nobert_rec13m/view.html