René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Optimize file I/O tools in llm-coding-tools-core","articleBody":"## Summary\n\n- Benchmark all 5 core file I/O tools (read, edit, write, glob, grep) with criterion using **real Rust source files** as test data\n- Optimize `read_file` hot path: 35-75% faster across all file sizes\n- Optimize `GrepOutput::format()`: 69-76% faster for both line-number modes\n- Fix arithmetic overflow in `read_file` capacity estimation\n- Edit, write, glob, and grep search are I/O-bound; algorithmic improvements yield 0-4%\n\n## Commits\n\nSplit into two commits:\n\n1. **Add criterion benchmarks** - benchmark suites + corpus data only\n2. **Optimize file I/O tools** - source code optimizations\n\n## Realistic benchmark corpus\n\nBenchmark test data is sourced from real Rust files (Apache-2.0, [codex-rs](https://github.com/openai/codex)) selected to match aggregate statistics measured across ~3,300 `.rs` files:\n\n| Metric | Corpus | Measured Average |\n|--------|--------|-----------------|\n| Non-blank avg line length | 37.7 chars | 37.7 chars |\n| Blank line ratio | ~10.5% | ~10.5% |\n| Avg bytes/line | 34.7 | 34.7 |\n\n| Size | Source | Lines |\n|------|--------|-------|\n| Small | `core/src/tools/handlers/plan.rs` | 122 |\n| Medium | `mcp-server/src/outgoing_message.rs` | 356 |\n| Large | `core/src/unified_exec/session_manager.rs` | 770 |\n\nShared via `benches/common/mod.rs` with `#[path]` attribute (no Cargo.toml changes needed).\n\n## Key changes\n\n**read.rs** - The big win. Replaced `write!` format macro with manual `push_usize` integer formatting, added ASCII fast path with `from_utf8_unchecked`, made `process_line` const-generic over line_numbers, added bulk `memchr_iter` newline skip for offset reads, and used `from_utf8` direct check instead of `from_utf8_lossy` Cow wrapper. Fixed overflow in capacity math with saturating arithmetic.\n\n**grep.rs** - Applied same `write!` -\u003e `push_str` + `push_u64` transformation to `format()`, added `sort_unstable_by`, `to_str()` fast path for paths, and pre-allocated matches Vec.\n\n**edit.rs** - Combined `replace_all` into single-pass `match_indices` loop. I/O-bound at ~15-67 µs; further gains blocked by syscall overhead.\n\n**write.rs / glob.rs** - Minimal gains; dominated by filesystem I/O. Glob adds partial sort via `select_nth_unstable_by` and deferred String conversion. OverrideBuilder approach was tested and reverted due to nested `.gitignore` correctness bug.\n\n## Significant improvements\n\n| Tool | Case | Before | After | Change |\n|------|------|--------|-------|--------|\n| read | small (122 lines, with LN) | 2.2 µs | 1.42 µs | -35% |\n| read | medium (356 lines, with LN) | 14.1 µs | 6.6 µs | -53% |\n| read | large (770 lines, with LN) | 53.2 µs | 24.9 µs | -53% |\n| read | offset_read (with LN) | 2.5 µs | 1.68 µs | -33% |\n| read | crlf (356 lines, with LN) | 13.9 µs | 6.58 µs | -53% |\n| read | small (122 lines, no LN) | - | 1.25 µs | new |\n| read | medium (356 lines, no LN) | - | 4.7 µs | new |\n| read | large (770 lines, no LN) | - | 16.7 µs | new |\n| read | offset_read (no LN) | - | 1.48 µs | new |\n| read | crlf (356 lines, no LN) | - | 4.7 µs | new |\n| grep format | with line numbers (~300 matches) | 5.60 µs | 1.49 µs | -73% |\n| grep format | without line numbers (~300 matches) | 2.67 µs | 0.83 µs | -69% |\n\n## No significant improvements (I/O-bound)\n\n| Tool | Case | Before | After | Change | Bottleneck |\n|------|------|--------|-------|--------|------------|\n| edit | rename_string (122 lines) | 19.6 µs | 19.2 µs | -2% | 6 syscalls |\n| edit | replace_all (770 lines) | 66.6 µs | 65.4 µs | -2% | 6 syscalls |\n| edit | not_found (356 lines) | 13.0 µs | 13.0 µs | ~0% | 6 syscalls |\n| edit | update_constant (770 lines) | 14.9 µs | 14.6 µs | -2% | 6 syscalls |\n| write | small (~4 KB) | 17 µs | 17 µs | ~0% | mkdir + write |\n| write | medium (~12 KB) | 26 µs | 26 µs | ~0% | write syscall |\n| write | nested_dirs (a/b/c) | 42 µs | 41 µs | ~0% | mkdir -p |\n| glob | small_tree (~10 files) | 104 µs | 102 µs | ~0% | walk + stat |\n| glob | large_tree (~300 files) | 944 µs | 950 µs | ~0% | walk + stat |\n| glob | no_matches (~300 files) | 578 µs | 555 µs | -4% | walk + stat |\n| glob | deep_nesting (10 levels) | 146 µs | 141 µs | -3% | walk + stat |\n| grep search | single_file | 58 µs | 58 µs | ~0% | regex + I/O |\n| grep search | multi_file (10 files) | 79 µs | 76 µs | -4% | regex + I/O |\n| grep search | no_matches | 74 µs | 71 µs | -4% | regex + I/O |\n| grep search | regex_pattern (fn\\s+\\w+) | 2.4 ms | 2.4 ms | ~0% | regex engine |\n| grep search | large_tree (30 files) | 241 µs | 236 µs | -2% | regex + I/O |\n\nedit, write, glob, and grep search are dominated by kernel VFS syscall overhead and external crate costs (grep-regex, ignore walker). The algorithmic improvements are real but invisible under I/O noise.\n\n## Bugs fixed\n\n| Tool | Bug | Status |\n|------|-----|--------|\n| read | Arithmetic overflow in capacity estimation (`limit * 64` could panic/wrap) | Fixed: saturating arithmetic + 1 MiB cap |\n| glob | OverrideBuilder bypassed nested `.gitignore` rules | Reverted; kept partial-sort optimizations |\n\n## Benchmark suite overview\n\n### tools_read.rs - 5 cases × 2 modes (with/without line numbers)\n| Case | Corpus | What it tests |\n|------|--------|---------------|\n| small_file | Small (122 lines) | Small file, fits in one read |\n| medium_file | Medium (356 lines) | Medium file, buffered reads |\n| large_file | Large (770 lines) | Large file, many lines processed |\n| offset_read | Medium + offset 50 + limit 50 | Partial read with offset |\n| crlf_file | Medium with CRLF endings | CRLF stripping overhead |\n\n### tools_edit.rs - 5 cases\n| Case | Corpus | What it tests |\n|------|--------|---------------|\n| rename_string | Small | Rename `update_plan` → `update_task_plan` |\n| change_type | Medium | Change type `mpsc::UnboundedSender` → `mpsc::Sender` |\n| update_constant | Large | Update `Duration::from_millis(100)` → `200` |\n| replace_all | Large | Bulk rename `process_id` → `session_id` |\n| not_found | Medium | Error path: string not found |\n\n### tools_write.rs - 4 cases\n| Case | Content | What it tests |\n|------|---------|---------------|\n| small_write | Corpus Small (~4 KB) | Write small source file |\n| medium_write | Corpus Medium (~12 KB) | Write medium source file |\n| large_write | Corpus Large (~26 KB) | Write large source file |\n| nested_dirs | Corpus Small + a/b/c/ path | Create nested directories |\n\n### tools_grep.rs - 5 cases × 2 modes + 2 format benchmarks\n| Case | Files | Pattern | What it tests |\n|------|-------|---------|---------------|\n| single_file | 1 | `fn ` | Single file, function definitions |\n| multi_file | 10 | `fn ` | Multi-file search |\n| no_matches | 10 | `xyznonexistent` | Fast rejection |\n| regex_pattern | 10 | `fn\\s+\\w+` | Complex regex matching |\n| large_tree | 30 | `fn ` | Large directory tree traversal |\n\n### tools_glob.rs - 4 cases\n| Case | Files | Pattern | What it tests |\n|------|-------|---------|---------------|\n| small_tree | ~10 | `**/*.rs` | Small project |\n| large_tree | ~300 | `**/*.rs` | Large monorepo |\n| no_matches | ~300 | `*.xyz` | Full traversal, no matches |\n| deep_nesting | ~10 | `**/*.rs` | Deep directory nesting |\n\n## Test plan\n\n- [x] All 357 existing tests pass across all feature flags (tokio, blocking, linux-bubblewrap)\n- [x] `.cargo/verify.sh` passes (build, test, clippy, docs, fmt)\n- [x] Edge-case tests for all bug fixes\n- [x] 5 new benchmark suites compile and produce valid throughput measurements","author":{"url":"https://github.com/Sewer56","@type":"Person","name":"Sewer56"},"datePublished":"2026-04-08T14:23:54.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":2},"url":"https://github.com/96/ReloadedCode/issues/96"}