René's URL Explorer Experiment

Title: sqlite: add batched async Database API by BurningEnlightenment · Pull Request #62015 · nodejs/node · GitHub

Open Graph Title: sqlite: add batched async Database API by BurningEnlightenment · Pull Request #62015 · nodejs/node

X Title: sqlite: add batched async Database API by BurningEnlightenment · Pull Request #62015 · nodejs/node

Description: This PR adds the asynchronous complements Database and Statement to the node:sqlite module. Given that a single sqlite db connection cannot scale beyond a single CPU core, (unlike the old sqlite3) this implementation avoids unordered dispatches to the thread pool by maintaining a strict FIFO operation queue. This allows usage of multiple db connections in parallel without relying on the internal db connection mutex (=> SQLITE_THREADSAFE=2 is sufficient). Furthermore I've favored throughput over latency, e.g. it collects asynchronous operations on a db connection and dispatches them in batches to the thread pool. This helps to amortize the inherent synchronization overhead over multiple operations and it also improves cache locality. The implementation effort has reached a maturity level where the overall approach should only require relatively small incremental changes to reach the outlined API scope. Briefly the following things work: Opening a db connection and (asynchronously) preparing statements. Disposal of db connection and statement instances.1 Database.prototype.exec() without a prepared statement. Statement.prototype.get()|all()|run() with binding parameters (though currently not as fancy as the sync variant). Implicit microtask-based batching of all previously mentioned operations (inspired by kysely). Things which still need work: Function parameter validation should not throw but return rejected promises. Implementing the remaining APIs: Database#location(dbName?: string): Promise Database#loadExtension(path: string): Promise; Database#enableLoadExtension(allow: boolean): Promise Database#enableDefensive(active: boolean): Promise Probably a few extensions to the test suite here and there. Proper API documentation From a brief look it seems that direct usage of v8::Promise looses the async context/callstack. I need some guidance on how to fix that. Potential follow-up improvements: The memory allocation strategy should be tweaked to use std::pmr::monotonic_buffer_resource in a few places. An explicit batching API, either callback- or Symbol.dispose/using-based Explore an API to move a db connection from Database to DatabaseSync and vice versa Fixes #54307 Previous attempt #59109 (I've only taken the test suites from there). API synopsis interface StatementOptions { readBigInts?: boolean; returnArrays?: boolean; // NOT GOING TO BE IMPLEMENTED HERE: // allowBareNamedParameters // allowUnknownNamedParameters } interface DatabaseOpenOptions extends StatementOptions { open?: boolean; readOnly?: boolean; enableForeignKeyConstraints?: boolean; enableDoubleQuotedStringLiterals?: boolean; allowExtension?: boolean; timeout?: number; defensive?: boolean; limits?: object; } interface DatabaseCtor { new(path: string, options?: DatabaseOpenOptions): Database; } interface Database { open(): void; isOpen: boolean; close(): Promise; [Symbol.asyncDispose](): Promise; location(dbName?: string): Promise; loadExtension(path: string): Promise; enableLoadExtension(allow: boolean): Promise; enableDefensive(active: boolean): Promise; isInTransaction(): Promise; exec(sql: string): Promise; prepare(sql: string, options?: StatementOptions): Promise; } type Literal = null | number | bigint | string | ArrayBufferView; type BindParams = Record | Array; type RowResult = Record | Array; interface Statement { [Symbol.dispose](): void; get(params?: BindParams): Promise; all(params?: BindParams): Promise; run(params?: BindParams): Promise<{ changes: number, lastInsertRowid: number }>; } Developer's Certificate of Origin 1.1 By making a contribution to this project, I certify that: (a) The contribution was created in whole or in part by me and I have the right to submit it under the open source license indicated in the file; or (b) The contribution is based upon previous work that, to the best of my knowledge, is covered under an appropriate open source license and I have the right under that license to submit that work with modifications, whether created in whole or in part by me, under the same open source license (unless I am permitted to submit under a different license), as indicated in the file; or (c) The contribution was provided directly to me by some other person who certified (a), (b) or (c) and I have not modified it. (d) I understand and agree that this project and the contribution are public and that a record of the contribution (including all personal information I submit with it, including my sign-off) is maintained indefinitely and may be redistributed consistent with this project or the open source license(s) involved. Footnotes I've followed the spirit of DEP0137; should probably be reconciled with the sync API at some point. ↩

Open Graph Description: This PR adds the asynchronous complements Database and Statement to the node:sqlite module. Given that a single sqlite db connection cannot scale beyond a single CPU core, (unlike the old sqlite3) ...

X Description: This PR adds the asynchronous complements Database and Statement to the node:sqlite module. Given that a single sqlite db connection cannot scale beyond a single CPU core, (unlike the old sqlite3) ...

Opengraph URL: https://github.com/nodejs/node/pull/62015

X: @github

direct link

Domain: github.com

Links:

Viewport: width=device-width