René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Some of the step tasks have been OOM Killed.","articleBody":"I am facing \"oom_kill event in StepId=866679.batch. Some of the step tasks have been OOM Killed.\" while using avg_confidence strategy for my multilabel dataset with around 38000 images of size 224. I use torch Dataloader with batch size 8 to load the data. Here's a snippet of the code covering Active Learning loop -\r\n\r\nn_queries = 14\r\nfor i in range(n_queries):\r\n    if i == 0:\r\n        n_instances = 8\r\n    else:\r\n        power += 0.25\r\n        n_instances = batch(int(np.ceil(np.power(10, power))), batch_size)\r\n    total_samples += n_instances\r\n    n_instances_list.append(total_samples)\r\n    \r\n    print(f\"\\nQuery {i + 1}: Requesting {n_instances} samples.\")\r\n    print(f\"Number of samples in pool before query: {X_pool.shape[0]}\")\r\n\r\n    \r\n\r\n    with torch.device(\"cpu\"):\r\n        query_idx, _ = learner.query(X_pool, n_instances=n_instances) \r\n        query_idx = np.unique(query_idx)\r\n        query_idx = np.array(query_idx).flatten() \r\n\r\n    # Extract the samples based on the query indices\r\n    X_query = X_pool[query_idx]\r\n    y_query = y_pool[query_idx]\r\n    filenames_query = [filenames_pool[idx] for idx in query_idx]\r\n\r\n    print(\"Shape of X_query after indexing:\", X_query.shape)\r\n\r\n    if X_query.ndim != 4:\r\n        raise ValueError(f\"Unexpected number of dimensions in X_query: {X_query.ndim}\")\r\n    if X_query.shape[1:] != (224, 224, 3):\r\n        raise ValueError(f\"Unexpected shape in X_query dimensions: {X_query.shape}\")\r\n\r\n    X_cumulative = np.vstack((X_cumulative, X_query))\r\n    y_cumulative = np.vstack((y_cumulative, y_query))\r\n    filenames_cumulative.extend(filenames_query)\r\n\r\n    save_checkpoint(i + 1, X_cumulative, y_cumulative, filenames_cumulative, save_dir)\r\n\r\n    learner.teach(X=X_cumulative, y=y_cumulative)\r\n\r\n    y_pred = learner.predict(X_test_np)\r\n    accuracy = accuracy_score(y_test_np, y_pred)\r\n    f1 = f1_score(y_test_np, y_pred, average='macro')\r\n    acc_test_data.append(accuracy)\r\n    f1_test_data.append(f1)\r\n\r\n    print(f\"Accuracy after query {i + 1}: {accuracy}\")\r\n    print(f\"F1 Score after query {i + 1}: {f1}\")\r\n\r\n\r\n    # Early stopping check\r\n    if f1 \u003e best_f1_score:\r\n        best_f1_score = f1\r\n        wait = 0  # reset the wait counter\r\n    else:\r\n        wait += 1  # increment the wait counter\r\n        if wait \u003e= patience:\r\n            print(\"Stopping early due to no improvement in F1 score.\")\r\n            break\r\n\r\n    # Remove queried instances from the pool\r\n    X_pool = np.delete(X_pool, query_idx, axis=0)\r\n    y_pool = np.delete(y_pool, query_idx, axis=0)\r\n    filenames_pool = [filename for idx, filename in enumerate(filenames_pool) if idx not in query_idx]\r\n    print(f\"Number of samples in pool after query: {X_pool.shape[0]}\")\r\n\r\nThis code runs well till 11 iterations but in the 12th iteration I get the OOM kill error. \r\n\r\nI am using A100 GPU with 40GB RAM which should be sufficient for this loop. Could you please help me identify what could be going wrong which leads to excessive memory requirement. Is there a bottleneck in my code that I should address? Could it be the case that for every iterarion the data is held in the main memory and can it be freed somehow without breaking the code and distorting the results. ","author":{"url":"https://github.com/shubhamgp47","@type":"Person","name":"shubhamgp47"},"datePublished":"2024-07-27T13:07:40.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/189/modAL/issues/189"}