Title: csv: handle empty fields with skipinitialspace by widehyo1 · Pull Request #8304 · RustPython/RustPython · GitHub
Open Graph Title: csv: handle empty fields with skipinitialspace by widehyo1 · Pull Request #8304 · RustPython/RustPython
X Title: csv: handle empty fields with skipinitialspace by widehyo1 · Pull Request #8304 · RustPython/RustPython
Description: Closes #xxxx This PR follows our AI policy Summary csv.reader(..., skipinitialspace=True) preprocessed each record by splitting it on the delimiter and trimming both ends of every piece. This had two problems: a field containing only spaces could produce an invalid Rust slice and panic; trailing spaces were removed even though only initial spaces should be ignored; delimiters inside quoted fields were treated as field boundaries, so valid quoted spaces could be removed before csv-core parsed the record. For example, the old preprocessing could change: 'a, "b, c", d' before passing it to csv-core, removing the space inside "b, c". The raw split-and-join preprocessing is now removed. A small per-item quote scanner classifies field starts, quote boundaries, doubled quotes, escapes, delimiters, and record terminators. The skipinitialspace preprocessing discards only actual initial-space events and copies every other source byte unchanged. The same scanner is consumed by the custom quote-mode reader, avoiding separate implementations of the quote and escape transitions. QUOTE_NOTNULL and QUOTE_STRINGS The CPython test also covers QUOTE_NOTNULL and QUOTE_STRINGS. For these modes, an empty unquoted field must become None, while a quoted empty field must remain an empty string: ',""' -> [None, ''] Both fields have empty decoded byte contents, so the conversion cannot be decided from the final field buffer alone. The reader must retain whether each field began with a quote. The existing custom QUOTE_NONE reader is therefore extended into a shared reader path for: QUOTE_NONE with an escape character; QUOTE_NOTNULL; QUOTE_STRINGS. Each parsed field carries its bytes and a was_quoted flag. The flag is used only when QUOTE_NOTNULL or QUOTE_STRINGS converts an empty unquoted field to None. This avoids separate, duplicated functions for the two modes while preserving the existing QUOTE_NONE escape behavior. Test_Csv.test_read_skipinitialspace is unmarked now that its regular, QUOTE_NOTNULL, and QUOTE_STRINGS cases pass. Scope This is intentionally a narrow reader fix. It does not introduce a complete CPython-style CSV state machine and does not change: strict parsing; records spanning multiple iterator items; multiline reader lifecycle; complete QUOTE_STRINGS numeric conversion; writer quoting or lineterminator handling. Testing temp/workspace/before-commit-csv.sh passed, including: cargo fmt --check cargo run --release Lib/test/test_csv.py 128 tests passed 6 skipped 21 expected failures cargo run -- extra_tests/snippets/stdlib_csv.py cargo clippy -p rustpython-stdlib --all-targets the configured workspace test suite related issue #8310 Summary by CodeRabbit Summary by CodeRabbit Bug Fixes Improved CSV reader behavior for quoted fields, including correct handling of escaped quotes, delimiters, and record/line termination. When skipinitialspace=True, initial-space trimming is now quote-aware, preserving spaces inside quoted fields. Empty unquoted fields and empty quoted fields are now handled distinctly (null vs empty string, as applicable). Refined parsing behavior when escapechar is configured. Tests Added a regression test for skipinitialspace=True ensuring spaces inside quoted fields are preserved.
Open Graph Description: Closes #xxxx This PR follows our AI policy Summary csv.reader(..., skipinitialspace=True) preprocessed each record by splitting it on the delimiter and trimming both ends of every piece. This had...
X Description: Closes #xxxx This PR follows our AI policy Summary csv.reader(..., skipinitialspace=True) preprocessed each record by splitting it on the delimiter and trimming both ends of every piece. This had...
Opengraph URL: https://github.com/RustPython/RustPython/pull/8304
X: @github
Domain: github.com
| route-pattern | /:user_id/:repository/pull/:id/files(.:format) |
| route-controller | pull_requests |
| route-action | files |
| fetch-nonce | v2:bfc2c1f4-9cb7-5f47-4aca-3214539feecc |
| current-catalog-service-hash | ae870bc5e265a340912cde392f23dad3671a0a881730ffdadd82f2f57d81641b |
| request-id | E040:3A432B:43FDB36:5F5E5FE:6A5FB73F |
| html-safe-nonce | ea9bbb53b9dd90e02f9939e6963f57c16f5147e49e49680153eb8d46079109d9 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJFMDQwOjNBNDMyQjo0M0ZEQjM2OjVGNUU1RkU6NkE1RkI3M0YiLCJ2aXNpdG9yX2lkIjoiMzg0OTk4MTAzOTIxNDU3MzM3NiIsInJlZ2lvbl9lZGdlIjoiaWFkIiwicmVnaW9uX3JlbmRlciI6ImlhZCJ9 |
| visitor-hmac | c89250c9a93b1d25222ef3e8396e24116f656e59a80050ad021a0a61f09d93e8 |
| hovercard-subject-tag | pull_request:4076407382 |
| github-keyboard-shortcuts | repository,pull-request-list,pull-request-conversation,pull-request-files-changed,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/RustPython/RustPython/pull/8304/files |
| twitter:image | https://avatars.githubusercontent.com/u/87641474?s=400&v=4 |
| twitter:card | summary_large_image |
| og:image | https://avatars.githubusercontent.com/u/87641474?s=400&v=4 |
| og:image:alt | Closes #xxxx This PR follows our AI policy Summary csv.reader(..., skipinitialspace=True) preprocessed each record by splitting it on the delimiter and trimming both ends of every piece. This had... |
| og:site_name | GitHub |
| og:type | object |
| hostname | github.com |
| expected-hostname | github.com |
| None | e0cfc367faf9c4c774d2a903897aa214c786247c4f45ffcee48cda2b819fb0c8 |
| turbo-cache-control | no-preview |
| diff-view | unified |
| go-import | github.com/RustPython/RustPython git https://github.com/RustPython/RustPython.git |
| octolytics-dimension-user_id | 39710557 |
| octolytics-dimension-user_login | RustPython |
| octolytics-dimension-repository_id | 135201145 |
| octolytics-dimension-repository_nwo | RustPython/RustPython |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 135201145 |
| octolytics-dimension-repository_network_root_nwo | RustPython/RustPython |
| turbo-body-classes | logged-out env-production page-responsive full-width |
| disable-turbo | true |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | 94569c859bc88d6bf7de7e0ce3f2a11cd529a131 |
| ui-target | full |
| theme-color | #1e2327 |
| color-scheme | light dark |
Links:
Viewport: width=device-width