René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Active Learning Yields Poor Results in Multi-Label Task","articleBody":"I am using modAL for an active learning project in multi-label classification. My implementation is in PyTorch, and I use DinoV2 as the backbone model.\r\n For the same dataset, I apply both active learning (using minimum confidence and average confidence strategies) and random sampling. I select the same number of samples in both strategies, but the results from random sampling are significantly better than those from the active learning approach. I would like to know if this discrepancy might be due to an issue with my code or the modAL library's handling of multi-label classification. Below is my active learning loop:\r\n\r\n```\r\nfor i in range(n_queries):\r\n    if i == 12:\r\n        n_instances = X_pool.shape[0]\r\n    else:\r\n        n_instances = batch(int(np.ceil(np.power(10, POWER))), BATCH_SIZE)\r\n\r\n    print(f\"\\nQuery {i + 1}: Requesting {n_instances} samples from a pool of size {X_pool.shape[0]}\")\r\n\r\n    if X_pool.shape[0] \u003c n_instances:\r\n        print(\"Not enough samples left in the pool to query the desired number of instances.\")\r\n        break\r\n\r\n    query_idx, _ = learner.query(X_pool, n_instances=n_instances)\r\n    query_idx = np.unique(query_idx)\r\n\r\n    if len(query_idx) == 0:\r\n        print(\"No indices were selected, which may indicate an issue with the query function or pool.\")\r\n        continue\r\n\r\n    # Add the newly selected samples to the cumulative training set\r\n    cumulative_X_train.append(X_pool[query_idx])\r\n    cumulative_y_train.append(y_pool[query_idx])\r\n\r\n    # Concatenate all the samples to form the cumulative training data\r\n    X_train_cumulative = np.concatenate(cumulative_X_train, axis=0)\r\n    y_train_cumulative = np.concatenate(cumulative_y_train, axis=0)\r\n\r\n    learner.teach(X_train_cumulative, y_train_cumulative)\r\n\r\n    # Log the selected sample names\r\n    selected_sample_names = train_df.loc[query_idx, \"image\"].tolist()\r\n    print(f\"Selected samples in Query {i + 1}: {selected_sample_names}\")\r\n    with open(samples_log_file, mode='a', newline='') as f:\r\n        writer = csv.writer(f)\r\n        writer.writerow([i + 1] + selected_sample_names)\r\n\r\n    # Remove the selected samples from the pool\r\n    X_pool = np.delete(X_pool, query_idx, axis=0)\r\n    y_pool = np.delete(y_pool, query_idx, axis=0)\r\n\r\n    # Evaluate the model\r\n    y_pred = learner.predict(X_test_np)\r\n    accuracy = accuracy_score(y_test_np, y_pred)\r\n    f1 = f1_score(y_test_np, y_pred, average='macro')\r\n    acc_test_data.append(accuracy)\r\n    f1_test_data.append(f1)\r\n    print(f\"Accuracy after query {i + 1}: {accuracy}\")\r\n    print(f\"F1 Score after query {i + 1}: {f1}\")\r\n\r\n    # Early stopping logic\r\n    if f1 \u003e best_f1_score:\r\n        best_f1_score = f1\r\n        wait = 0\r\n    else:\r\n        wait += 1\r\n        if wait \u003e= patience:\r\n            print(f\"Stopping early after {i + 1} queries due to no improvement in F1 score.\")\r\n            break\r\n\r\n    total_samples += len(query_idx)\r\n    print(f\"Total samples used for training after query {i + 1}: {total_samples}\")\r\n    POWER += 0.25\r\n    torch.cuda.empty_cache()\r\n\r\n```\r\n   ","author":{"url":"https://github.com/shadikhamsehh","@type":"Person","name":"shadikhamsehh"},"datePublished":"2024-09-10T20:16:12.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/191/modAL/issues/191"}