René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Root Allocator doesn't release memory when its closed","articleBody":"### Describe the bug, including details regarding any error messages, version, and platform.\n\nThere is a bug in the root allocator where it doesn't release the memory when it's closed. Attached the sample code that prints the RSS memory. \n\nI ran this code with the following jvm params\n```\n--add-opens=java.base/java.nio=ALL-UNNAMED -Xms1g -Xmx1g -XX:+AlwaysPreTouch\n```\n\nCode\n```\npublic final class ArrowOnlyParquetMemoryRepro {\n\n  private static final long ARROW_MEMORY_LIMIT_BYTES = 100L * 1024L * 1024L; // 100 MB\n  private static final int ARROW_BATCH_SIZE_ROWS = 1024 * 1024; // 1 million rows\n\n  private ArrowOnlyParquetMemoryRepro() {\n  }\n\n  public static void main(String[] args) throws Exception {\n    // Parse arguments\n    File parquetFile = null;\n    List\u003cString\u003e scanCols = null;\n\n    for (int i = 0; i \u003c args.length; i++) {\n      if (\"--path\".equals(args[i]) \u0026\u0026 i + 1 \u003c args.length) {\n        parquetFile = new File(args[i + 1]);\n      } else if (\"--scanCols\".equals(args[i]) \u0026\u0026 i + 1 \u003c args.length) {\n        scanCols = parseCsv(args[i + 1]);\n      }\n    }\n\n    if (parquetFile == null || scanCols == null || scanCols.isEmpty()) {\n      System.err.println(\"Usage: ArrowOnlyParquetMemoryRepro --path \u003cfile.parquet\u003e --scanCols \u003ccol1,col2,...\u003e\");\n      System.exit(1);\n    }\n\n    if (!parquetFile.exists() || !parquetFile.isFile()) {\n      System.err.println(\"File does not exist: \" + parquetFile.getAbsolutePath());\n      System.exit(1);\n    }\n\n    log(\"config\", \"file=\" + parquetFile.getAbsolutePath() + \" cols=\" + scanCols);\n    logRss(\"start\");\n\n    // Scan each column\n    for (String col : scanCols) {\n      logRss(\"before_scan:\" + col);\n      scanColumn(parquetFile, col);\n      logRss(\"after_scan:\" + col);\n    }\n\n    logRss(\"end\");\n\n    // Force GC and observe RSS\n    System.gc();\n    Thread.sleep(1000);\n    logRss(\"after_gc\");\n\n    // Keep process alive for external memory inspection (e.g., via `ps` or `/proc/\u003cpid\u003e/status`)\n    log(\"info\", \"Process will now sleep. Use 'ps -o rss,pid,command' or 'cat /proc/\u003cpid\u003e/status' to inspect RSS.\");\n    while (true) {\n      Thread.sleep(60_000);\n      System.gc();\n      logRss(\"sleeping\");\n    }\n  }\n\n  private static void scanColumn(File parquetFile, String col) throws Exception {\n    long rowsScanned = 0;\n\n    try (RootAllocator allocator = new RootAllocator(ARROW_MEMORY_LIMIT_BYTES);\n        DatasetFactory datasetFactory = new FileSystemDatasetFactory(\n            allocator, NativeMemoryPool.getDefault(), FileFormat.PARQUET, parquetFile.toURI().toString());\n        Dataset dataset = datasetFactory.finish()) {\n\n      ScanOptions options = new ScanOptions.Builder(ARROW_BATCH_SIZE_ROWS)\n          .columns(Optional.of(new String[]{col}))\n          .build();\n\n      try (Scanner scanner = dataset.newScan(options);\n          ArrowReader reader = scanner.scanBatches()) {\n        while (reader.loadNextBatch()) {\n          VectorSchemaRoot root = reader.getVectorSchemaRoot();\n          if (root == null || root.getFieldVectors().isEmpty()) {\n            continue;\n          }\n          FieldVector vector = root.getVector(col);\n          if (vector == null) {\n            log(\"warn\", \"Column not found in file: \" + col);\n            return;\n          }\n          int rowCount = root.getRowCount();\n          // Access values to ensure they're materialized\n          for (int i = 0; i \u003c rowCount; i++) {\n            vector.getObject(i);\n          }\n          rowsScanned += rowCount;\n        }\n      }\n    }\n\n    log(\"scan\", \"col=\" + col + \" rowsScanned=\" + rowsScanned);\n  }\n\n  private static List\u003cString\u003e parseCsv(String csv) {\n    List\u003cString\u003e result = new ArrayList\u003c\u003e();\n    for (String part : csv.split(\",\")) {\n      String trimmed = part.trim();\n      if (!trimmed.isEmpty()) {\n        result.add(trimmed);\n      }\n    }\n    return result;\n  }\n\n  private static void logRss(String phase) {\n    long rssKb = readRssKb();\n    if (rssKb \u003c 0) {\n      log(\"rss\", \"phase=\" + phase + \" rssKb=N/A (not on Linux)\");\n    } else {\n      log(\"rss\", \"phase=\" + phase + \" rssKb=\" + rssKb + \" rssMb=\" + (rssKb / 1024));\n    }\n  }\n\n  private static long readRssKb() {\n    File status = new File(\"/proc/self/status\");\n    if (!status.exists()) {\n      return -1;\n    }\n    try (BufferedReader br = new BufferedReader(new FileReader(status, StandardCharsets.UTF_8))) {\n      String line;\n      while ((line = br.readLine()) != null) {\n        if (line.startsWith(\"VmRSS:\")) {\n          String[] parts = line.trim().split(\"\\\\s+\");\n          if (parts.length \u003e= 2) {\n            return Long.parseLong(parts[1]);\n          }\n        }\n      }\n      return -1;\n    } catch (Exception e) {\n      return -1;\n    }\n  }\n\n  private static void log(String tag, String msg) {\n    System.out.println(\"[\" + tag + \"] \" + msg);\n  }\n}\n```\n\nThis is the output\n```\n[rss] phase=start rssKb=1121340 rssMb=1095\n[rss] phase=before_scan:col_name rssKb=1121808 rssMb=1095\n2026-01-08T10:58:01,040 INFO  [main] org.apache.arrow.memory.BaseAllocator - Debug mode disabled. Enable with the VM option -Darrow.memory.debug.allocator=true.\n2026-01-08T10:58:01,049 INFO  [main] org.apache.arrow.memory.DefaultAllocationManagerOption - allocation manager type not specified, using netty as the default type\n2026-01-08T10:58:01,076 INFO  [main] org.apache.arrow.memory.CheckAllocator - Using DefaultAllocationManager at \u003cclass name\u003e\n[scan] col=col_name rowsScanned=2734751\n[rss] phase=after_scan:col_name rssKb=1313364 rssMb=1282\n[rss] phase=before_scan:col_name rssKb=1313364 rssMb=1282\n[scan] col=col_name rowsScanned=2734751\n[rss] phase=after_scan:col_name rssKb=1331144 rssMb=1299\n[rss] phase=before_scan:col_name rssKb=1331144 rssMb=1299\n[scan] col=col_name rowsScanned=2734751\n[rss] phase=after_scan:col_name rssKb=1338228 rssMb=1306\n[rss] phase=end rssKb=1338228 rssMb=1306\n[rss] phase=after_gc rssKb=1338228 rssMb=1306\n[info] Process will now sleep. Use 'ps -o rss,pid,command' or 'cat /proc/\u003cpid\u003e/status' to inspect RSS.\n[rss] phase=sleeping rssKb=1338228 rssMb=1306\n```","author":{"url":"https://github.com/krishan1390","@type":"Person","name":"krishan1390"},"datePublished":"2026-01-08T11:02:08.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":6},"url":"https://github.com/950/arrow-java/issues/950"}