Title: stdlib stubs are unnecessarily strict with file-like objects · Issue #4212 · python/typeshed · GitHub
Open Graph Title: stdlib stubs are unnecessarily strict with file-like objects · Issue #4212 · python/typeshed
X Title: stdlib stubs are unnecessarily strict with file-like objects · Issue #4212 · python/typeshed
Description: Problem Currently the IO situation is less than ideal. Not only are IO[str]/TextIO and IO[bytes]/BinaryIO a bit confusing (interchangeable in most cases), but the use of IO through stdlib is inconsistent and doing things like passing an ...
Open Graph Description: Problem Currently the IO situation is less than ideal. Not only are IO[str]/TextIO and IO[bytes]/BinaryIO a bit confusing (interchangeable in most cases), but the use of IO through stdlib is incons...
X Description: Problem Currently the IO situation is less than ideal. Not only are IO[str]/TextIO and IO[bytes]/BinaryIO a bit confusing (interchangeable in most cases), but the use of IO through stdlib is incons...
Opengraph URL: https://github.com/python/typeshed/issues/4212
X: @github
Domain: github.com
{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"stdlib stubs are unnecessarily strict with file-like objects","articleBody":"## Problem\r\n\r\nCurrently the IO situation is less than ideal. Not only are `IO[str]`/`TextIO` and `IO[bytes]`/`BinaryIO` a bit confusing (interchangeable in most cases), but the use of `IO` through stdlib is inconsistent and doing things like passing an object with a `write()` method to `json.dump()` does not work.\r\n\r\nThis is because the `IO` object, while describing the actual objects returned by `open()` perfectly, is not suitable to represent the \"file-like object\" interface. This interface is well known, documented prominently in the standard library's documentation ([glossary: \"file object\" and \"file-like object\"](https://docs.python.org/3/glossary.html#term-file-object)) and a testament to duck-typing; however it's not compatible with how typeshed is currently written (for the most part).\r\n\r\n## Proposal\r\n\r\nI propose to introduce `Protocol`s (not abstract classes) to be used for parameters where a \"file object\" is expected, allowing one to correctly type their file-like objects without having to inherit one of the abstract base classes. Furthermore, I think we should have two protocols representing files that can be read from or written to.\r\n\r\nThis work can be done incrementally, and I am willing to spend time doing this if there is no veto to this ticket.\r\n\r\n## Pros\r\n\r\nThis would allow a file-like object to be passed to `json.dump()`, `zipfile.ZipFile`, and others (like it already can to `csv.write()`).\r\n\r\nUsing `Protocol`s of this small scale would allow objects that already conform to be used in interfaces expecting file-like object, without having to implement too many methods (or explicitly inherit from the base class, as is required now). This should lower the effort of bringing libraries to the typing world. Using two separate protocols is similar to how most languages do this, off the top of my head:\r\n\r\n* Rust ([`io::Read`](https://doc.rust-lang.org/nightly/std/io/trait.Read.html) and [`io::Write`](https://doc.rust-lang.org/nightly/std/io/trait.Write.html) traits).\r\n* Java (`InputStream` with `read()`, `OutputStream` with `write()` and `flush()`)\r\n* C++ ([`istream`](http://www.cplusplus.com/reference/istream/istream/), [`ostream`](http://www.cplusplus.com/reference/ostream/ostream/))\r\n\r\nIt is interesting to note that the protocols I describe already exist in `typeshed`. Not wanting to put `IO` where the documentation called for `file-like object`, protocols have already been introduced:\r\n\r\n* for `shutil.copyfileobj()`: [`_Reader` and `_Writer` protocols](https://github.com/python/typeshed/blob/2bd06a0a3dd0a37127484e3b543c232cc0fc3577/stdlib/2and3/shutil.pyi#L46-L50)\r\n* for `csv.writer()`: [`_csv._Writer` protocol](https://github.com/python/typeshed/blob/afe1e543b3433fa821500b34f79c620110262f01/stdlib/2and3/_csv.pyi#L44-L45)\r\n\r\nIntroducing those protocols would also allow us to remove some of the `IO[str]`/`TextIO` complexity: while `TextIO` and `BinaryIO` are still needed for the native file objects (they have additional methods compared to `IO`), the protocols used for function parameters everywhere can be only `Read[str]` and `Read[bytes]`.\r\n\r\n## Cons\r\n\r\nThis is a sizeable change, and people are likely to use both the base class and those protocols for some time. However code using the base class should not break when passed to functions expecting the protocol.\r\n\r\nAnother caveat is that this might give a false sense of security: libraries in the wild do their own check to determine if an object conform to the interface, and for example `pandas` will not accept to write on a file object that does implement `__iter__`. Therefore objects conforming to the protocol might still not be accepted by (IMHO buggy) libraries, while inheriting the base class would make their objects look more like file objects (maybe too much, since it gives everything the attributes of both a readable and a writable file!).\r\n\r\n## Draft\r\n\r\nUnfortunately this is where the typeshed-bikeshed starts, but this is my proposal:\r\n\r\n```python\r\nAnyStr = typing.TypeVar('AnyStr', str, bytes) # typing.AnyStr\r\n\r\nclass WriteIO(Protocol[AnyStr]):\r\n def write(self, s: AnyStr) -> int: ...\r\n def flush(self) -> None: ...\r\n def close(self) -> None: ...\r\n def __enter__(self) -> 'WriteIO[AnyStr]': ...\r\n def __exit__(self, exc_type: Optional[Type[BaseException]], exc_val: Optional[BaseException],\r\n exc_tb: Optional[TracebackType]) -> Optional[bool]: ...\r\n\r\nclass ReadIO(Protocol[AnyStr]):\r\n def read(self, size: typing.Optional[int] = None) -> AnyStr: ...\r\n def close(self) -> None: ...\r\n def __enter__(self) -> 'WriteIO[AnyStr]': ...\r\n def __exit__(self, exc_type: Optional[Type[BaseException]], exc_val: Optional[BaseException],\r\n exc_tb: Optional[TracebackType]) -> Optional[bool]: ...\r\n def __iter__(self) -> Iterator[AnyStr]: ...\r\n```\r\n\r\nAdditional protocols can be added to provide `seek()`/`tell()` (similar to Rust's [io::Seek trait](https://doc.rust-lang.org/nightly/std/io/trait.Seek.html))","author":{"url":"https://github.com/remram44","@type":"Person","name":"remram44"},"datePublished":"2020-06-10T02:17:09Z","url":"https://github.com/python/typeshed/issues/4212"}
| react-profiling | 0 |
| react-import-map | react,react-dom,react-dom/client,react-dom/profiling,react-is,react/compiler-runtime,react/jsx-dev-runtime,react/jsx-runtime,scheduler@b8488ee2c322 |
| route-pattern | /_view_fragments/issues/show/:user_id/:repository/:id/issue_layout(.:format) |
| route-controller | voltron_issues_fragments |
| route-action | issue_layout |
| fetch-nonce | v2:e97e630f-a6be-6431-4c1a-6515cfd3db8d |
| current-catalog-service-hash | 81bb79d38c15960b92d99bca9288a9108c7a47b18f2423d0f6438c5b7bcd2114 |
| request-id | E616:196792:2ECF0AC:3E125DB:6B002BFD |
| html-safe-nonce | 0251c527c39d031185222117c8352fce8bf6a3d7693f6251741b6be328ddafc6 |
| visitor-payload | eyJyZWZlcnJlciI6IiIsInJlcXVlc3RfaWQiOiJFNjE2OjE5Njc5MjoyRUNGMEFDOjNFMTI1REI6NkIwMDJCRkQiLCJ2aXNpdG9yX2lkIjoiMzA5MTU4MTQyOTY3NTU3MTE3IiwicmVnaW9uX2VkZ2UiOiJpYWQiLCJyZWdpb25fcmVuZGVyIjoiaWFkIn0= |
| visitor-hmac | 7c7f2a986d0bbb90a456098da70c8d51bfc003989a1a9340b4d8540b01647841 |
| hovercard-subject-tag | issue:635874737 |
| github-keyboard-shortcuts | repository,issues,copilot |
| google-site-verification | Apib7-x98H0j5cPqHWwSMm6dNU4GmODRoqxLiDzdx9I |
| octolytics-url | https://collector.github.com/github/collect |
| analytics-location | / |
| fb:app_id | 1401488693436528 |
| apple-itunes-app | app-id=1477376905, app-argument=https://github.com/_view_fragments/issues/show/python/typeshed/4212/issue_layout |
| twitter:image | https://opengraph.githubassets.com/e9f0d4f35c138fa63544ec85f5e024319d84a821298e0fe5759a5510ac0b5f5d/python/typeshed/issues/4212 |
| twitter:card | summary_large_image |
| og:image | https://opengraph.githubassets.com/e9f0d4f35c138fa63544ec85f5e024319d84a821298e0fe5759a5510ac0b5f5d/python/typeshed/issues/4212 |
| og:image:alt | Problem Currently the IO situation is less than ideal. Not only are IO[str]/TextIO and IO[bytes]/BinaryIO a bit confusing (interchangeable in most cases), but the use of IO through stdlib is incons... |
| og:image:width | 1200 |
| og:image:height | 600 |
| og:site_name | GitHub |
| og:type | object |
| og:author:username | remram44 |
| hostname | github.com |
| expected-hostname | github.com |
| None | 7d0829bbde9e48b2486cb5dd6d147896eb01d32134627091f818958a935ef3f0 |
| turbo-cache-control | no-preview |
| go-import | github.com/python/typeshed git https://github.com/python/typeshed.git |
| octolytics-dimension-user_id | 1525981 |
| octolytics-dimension-user_login | python |
| octolytics-dimension-repository_id | 31696383 |
| octolytics-dimension-repository_nwo | python/typeshed |
| octolytics-dimension-repository_public | true |
| octolytics-dimension-repository_is_fork | false |
| octolytics-dimension-repository_network_root_id | 31696383 |
| octolytics-dimension-repository_network_root_nwo | python/typeshed |
| turbo-body-classes | logged-out env-production page-responsive |
| disable-turbo | false |
| browser-stats-url | https://api.github.com/_private/browser/stats |
| browser-errors-url | https://api.github.com/_private/browser/errors |
| release | f3f3a43ca095cdea3abbb394112e971f9fdb94fa |
| ui-target | canary-2 |
| theme-color | #1e2327 |
| color-scheme | light dark |
| react-app-name | issues |
Links:
Viewport: width=device-width