René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Class-typed fields are required when inserting a new document","articleBody":"### Initial Checks\n\n- [X] I have read and followed [the docs](https://docs.docarray.org/) and still think this is a bug\n\n### Description\n\nI have this dataclass:\r\n```python\r\nclass QuoteFile(BaseDoc):\r\n    quote_file_id: int = None\r\n    extracted_data: Optional[ExtractedDocumentData] = None\r\n```\r\n\r\nWhen indexing a new document, `extracted_data` is always required no matter what I do (type it with `Optional` etc). This is especially caused by `BaseDocIndex._get_values_by_column()`. \r\n\r\nI have finally forked `docarray`, will send a PR for various minor fixes like this one:\r\n\r\n\u003cimg width=\"729\" alt=\"image\" src=\"https://github.com/docarray/docarray/assets/656635/3c97aebf-a386-4cda-ada1-959b3a5a8fe9\"\u003e\r\n\r\nYou can see my changes here, the highlighted line returns None when the field object is optional and not given for a new document during indexing.\n\n### Example Code\n\n_No response_\n\n### Python, DocArray \u0026 OS Version\n\n```Text\nPython 3.8.12\r\ndocarray==0.40.0\n```\n\n\n### Affected Components\n\n- [X] [Vector Database / Index](https://docs.docarray.org/user_guide/storing/docindex/)\n- [ ] [Representing](https://docs.docarray.org/user_guide/representing/first_step)\n- [ ] [Sending](https://docs.docarray.org/user_guide/sending/first_step/)\n- [X] [storing](https://docs.docarray.org/user_guide/storing/first_step/)\n- [ ] [multi modal data type](https://docs.docarray.org/data_types/first_steps/)","author":{"url":"https://github.com/oytuntez","@type":"Person","name":"oytuntez"},"datePublished":"2024-02-08T14:42:40.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":3},"url":"https://github.com/1851/docarray/issues/1851"}