René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Memory consumption issue","articleBody":"I use the command:\r\n```\r\nbash run.sh train RotatE FB15k-237 0 0 1024 256 1000 9.0 1.0 0.00005 100000 16 -de\r\n```\r\nto train RotatE on a 11 GB GPU. I ensure it is completely free.\r\nI still get the following error:\r\n```\r\n2022-03-31 19:32:37,370 INFO     negative_adversarial_sampling = False\r\n2022-03-31 19:32:37,370 INFO     learning_rate = 0\r\n2022-03-31 19:32:39,079 INFO     Training average positive_sample_loss at step 0: 5.635527\r\n2022-03-31 19:32:39,079 INFO     Training average negative_sample_loss at step 0: 0.003591\r\n2022-03-31 19:32:39,079 INFO     Training average loss at step 0: 2.819559\r\n2022-03-31 19:32:39,079 INFO     Evaluating on Valid Dataset...\r\n2022-03-31 19:32:39,552 INFO     Evaluating the model... (0/2192)\r\n2022-03-31 19:33:38,650 INFO     Evaluating the model... (1000/2192)\r\n2022-03-31 19:34:38,503 INFO     Evaluating the model... (2000/2192)\r\n2022-03-31 19:34:49,981 INFO     Valid MRR at step 0: 0.005509\r\n2022-03-31 19:34:49,982 INFO     Valid MR at step 0: 6894.798660\r\n2022-03-31 19:34:49,982 INFO     Valid HITS@1 at step 0: 0.004733\r\n2022-03-31 19:34:49,982 INFO     Valid HITS@3 at step 0: 0.005076\r\n2022-03-31 19:34:49,982 INFO     Valid HITS@10 at step 0: 0.005646\r\nTraceback (most recent call last):\r\n  File \"codes/run.py\", line 371, in \u003cmodule\u003e\r\n    main(parse_args())\r\n  File \"codes/run.py\", line 315, in main\r\n    log = kge_model.train_step(kge_model, optimizer, train_iterator, args)\r\n  File \"/home/prachi/related_work/KnowledgeGraphEmbedding/codes/model.py\", line 315, in train_step\r\n    loss.backward()\r\n  File \"/home/prachi/anaconda3/envs/py36/lib/python3.6/site-packages/torch/tensor.py\", line 102, in backward\r\n    torch.autograd.backward(self, gradient, retain_graph, create_graph)\r\n  File \"/home/prachi/anaconda3/envs/py36/lib/python3.6/site-packages/torch/autograd/__init__.py\", line 90, in backward\r\n    allow_unreachable=True)  # allow_unreachable flag\r\nRuntimeError: CUDA out of memory. Tried to allocate 1.95 GiB (GPU 0; 10.92 GiB total capacity; 7.41 GiB already allocated; 1.51 GiB free; 1.52 GiB cached)\r\nrun.sh: line 79: \r\nCUDA_VISIBLE_DEVICES=$GPU_DEVICE python -u $CODE_PATH/run.py --do_train \\\r\n    --cuda \\\r\n    --do_valid \\\r\n    --do_test \\\r\n    --data_path $FULL_DATA_PATH \\\r\n    --model $MODEL \\\r\n    -n $NEGATIVE_SAMPLE_SIZE -b $BATCH_SIZE -d $HIDDEN_DIM \\\r\n    -g $GAMMA -a $ALPHA -adv \\\r\n    -lr $LEARNING_RATE --max_steps $MAX_STEPS \\\r\n    -save $SAVE --test_batch_size $TEST_BATCH_SIZE \\\r\n    ${14} ${15} ${16} ${17} ${18} ${19} ${20}\r\n\r\n: No such file or directory\r\n```\r\nI get similar errors on trying to train FB15k using the command in best_config.sh file.\r\nI reduced the batchsize to 500 and it worked but the performance is much less than the numbers reported in the paper.\r\n\r\nI am not sure what is the issue.","author":{"url":"https://github.com/p6jain","@type":"Person","name":"p6jain"},"datePublished":"2022-03-31T14:21:18.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":1},"url":"https://github.com/53/KnowledgeGraphEmbedding/issues/53"}