René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"ldap.initialize(fileno=fileno) does take ownership of the fileno","articleBody":"Issue description:\r\n\r\nThe [ldap.initialize() documents](https://github.com/python-ldap/python-ldap/blob/a58282adbc6b1f5f9755458227e6bb8667b72f6b/Doc/reference/ldap.rst?plain=1#L43) say\r\n\u003e If fileno parameter is given then the file descriptor will be used to connect to an LDAP server. The fileno must either be a socket file descriptor as [int](https://docs.python.org/3/library/functions.html#int) or a file-like object with a fileno() method that returns a socket file descriptor. The socket file descriptor must already be connected. [LDAPObject](https://www.python-ldap.org/en/python-ldap-3.4.3/reference/ldap.html#ldap.ldap.ldapobject.LDAPObject) **does not take ownership of the file descriptor**. It must be kept open during operations and explicitly closed after the [LDAPObject](https://www.python-ldap.org/en/python-ldap-3.4.3/reference/ldap.html#ldap.ldap.ldapobject.LDAPObject) is unbound.\r\n\r\nHowever it seemingly is the case that it *does* take ownership\r\n\r\nhttps://github.com/python-ldap/python-ldap/commit/1490e999e10960f0fdf974f6da804a2fc55e4b05, https://github.com/python-ldap/python-ldap/issues/460, https://github.com/python-ldap/python-ldap/pull/543 are likely due to this confusion as well.\r\n\r\nFollowing the code shows it goes like this.\r\n\r\n`ldap.initalize(*args, fileno=fd)` will result in `ldap_init_fd()` being called which executes [code](https://github.com/openldap/openldap/blob/c9ab732ec18e32c58643e44a45233a6ed87830b6/libraries/libldap/open.c#L329) that essentially creates a new ldap connection object that is then bound to the passed in fileno it then based on the protocol passed (LDAP_PROTO_TCP, LDAP_PROTO_UDP, LDAP_PROTO_IPC, LDAP_PROTO_EXT) will assign the protocols respective sockbuf_IO(s) set of {setup, ctrl, read, write, close} functions (except for LDAP_PROTO_EXT which the user specifies a custom struct elsewhere). These functions dictate how to interact with the underlying communication mechanism. Most relevant to this issue is the “close” functions\r\n\r\nFor TCP/UDP sockbuf_IO it calls “tcp_close()” which is system dependent macro but for Linux is likely  the [below](https://github.com/openldap/openldap/blob/c9ab732ec18e32c58643e44a45233a6ed87830b6/include/ac/socket.h#L142)\r\n\u003e define tcp_close( s )\t(shutdown( s, SHUT_RDWR ), close( s ))\r\n\r\nFor IPC it uses ber_sockbuf_io_fd (where sb_fd_close()  is just close(2)\r\n\r\nIn all cases except LDAP_PROTO_EXT(where it’s user dependent), when a connection is freed some form of close() is called on the fd.\r\n\r\nI’m not sure why the documentation thinks it doesn’t “take ownership”, but maybe it’s because of this section in ldap_init_fd()\r\n```\r\n\r\n\t/* Attach the passed socket as the LDAP's connection */\r\n\tconn = ldap_new_connection( ld, NULL, 1, 0, NULL, 0, 0 );\r\n\tif( conn == NULL ) {\r\n\t\tLDAP_MUTEX_UNLOCK( \u0026ld-\u003eld_conn_mutex );\r\n\t\tldap_unbind_ext( ld, NULL, NULL );\r\n\t\treturn( LDAP_NO_MEMORY );\r\n\t}\r\n\tif( url )\r\n\t\tconn-\u003elconn_server = ldap_url_dup( ld-\u003eld_options.ldo_defludp );\r\n\tber_sockbuf_ctrl( conn-\u003elconn_sb, LBER_SB_OPT_SET_FD, \u0026fd );\r\n\tld-\u003eld_defconn = conn;\r\n\t++ld-\u003eld_defconn-\u003elconn_refcnt;\t/* so it never gets closed/freed */\r\n```\r\n\r\nHowever this just means the connection won’t be reclaimed while LDAP object is in use, but that doesn’t stop it from being cleaned up on free.\r\n\r\nIf I’m following things correctly:\r\n`ldap.initialize(*args, fileno=fileno)` will create a ldap object with an ldap connection from the connected socket fd we pass in. \r\n\r\nBut on ldap_unbind_ext() which ldap_destroy(), ldap_unbind(), [`dealloc(LDAPObject *self)`](https://github.com/python-ldap/python-ldap/blob/a58282adbc6b1f5f9755458227e6bb8667b72f6b/Modules/LDAPObject.c#L33) etc, eventually call. `ldap_unbind_ext()` will call [`ldap_ld_free`](https://github.com/openldap/openldap/blob/c9ab732ec18e32c58643e44a45233a6ed87830b6/libraries/libldap/unbind.c#L74) which if the ref count for the ldap object is \u003c 2, it will perform\r\n```\r\n\t/* free and unbind from all open connections */\r\n\twhile ( ld-\u003eld_conns != NULL ) {\r\n\t\tldap_free_connection( ld, ld-\u003eld_conns, /*force=*/1, /*unbind=*/close);\r\n\t}\r\n```\r\n\r\nWell ldap_free_connection() when force=1, [will ignore the connection refcount](https://github.com/openldap/openldap/blob/c9ab732ec18e32c58643e44a45233a6ed87830b6/libraries/libldap/request.c#L752) and free the ldap connection anyway.\r\n\r\nSpecifically it frees/closes the fd by calling either of these functions\r\n\r\n```\r\nif ( lc-\u003elconn_sb != ld-\u003eld_sb ) {\r\n\t\t\tber_sockbuf_free( lc-\u003elconn_sb );\r\n\t\t} else {\r\n\t\t\tber_int_sb_close( lc-\u003elconn_sb );\r\n\t\t}\r\n```\r\n\r\nWhere `_sockbuf_free()` also calls [`ber_int_sb_close()`](https://github.com/openldap/openldap/blob/master/libraries/liblber/sockbuf.c#L375)\r\n\r\nAnd that function is \r\n```\r\nSockbuf_IO_Desc\t\t*p;\r\n\r\n\tassert( sb != NULL);\r\n   \r\n\tp = sb-\u003esb_iod;\r\n\twhile ( p ) {\r\n\t\tif ( p-\u003esbiod_io-\u003esbi_close \u0026\u0026 p-\u003esbiod_io-\u003esbi_close( p ) \u003c 0 ) {\r\n\t\t\treturn -1;\r\n\t\t}\r\n\t\tp = p-\u003esbiod_next;\r\n\t}\r\n   \r\n\tsb-\u003esb_fd = AC_SOCKET_INVALID;\r\n   \r\n\treturn 0;\r\n```\r\n\r\nGoing way back to the Sockbuf_IO functions that were bound during the initalize discussed above. \r\n\r\nWhich means for all the supported protocols in `python-ldap` (TCP, UDP, IPC) at minimum `close(fd)` is called.\r\n\r\nLDAPObject’s deconstructor will call `ldap_unbind_ext()` which as described above means it will always lead to closing the FD passed in to `ldap.initalize(*args, fileno=fileno)`\r\n\r\n[`ldap_unbind_ext(3)`](https://man7.org/linux/man-pages/man3/ldap.3.html) is defined as\r\n\u003e ldap_unbind_ext(3)\r\n              synchronously unbind from the LDAP server and close the\r\n              connection\r\nSo it seems like this is somewhat expected.\r\n\r\nWhy do the Docs believe libopenldap does not take ownership?\r\n\r\nSteps to reproduce: \r\nThe simplest repro is \r\n\r\n```\r\nimport ldap\r\nimport socket\r\n# set openldap debugging levels\r\nldap.set_option(ldap.OPT_DEBUG_LEVEL, 255)\r\n# monkey patch __del__ method to see when it is called\r\nldap.ldapobject.LDAPObject.__del__ = lambda self: print(f\"calling __del__ of {self}\")\r\n\r\ndef simple_repro():\r\n    # Connect to an \"LDAP server\" (that doesn't exist)\r\n    client, server = socket.socketpair(family=socket.AF_UNIX, type=socket.SOCK_STREAM)\r\n    print(f\"client={client.fileno()}, server={server.fileno()}\")\r\n    l = ldap.initialize('ldap://localhost:1389', trace_level=1, fileno=client.fileno())\r\n    print(f\"l's fileno={l.fileno()}\")\r\n    del l # force l's deconstruction\r\n    server.close()\r\n    # this will fail due to l's deconstruction resulting in close(client.fileno())\r\n    client.close()\r\n\r\nsimple_repro()\r\n```\r\n\r\nWhich outputs\r\n```\r\nclient=3, server=4\r\nldap_url_parse_ext(ldap://localhost:1389)\r\nldap_create\r\nldap_url_parse_ext(ldap://localhost:1389)\r\nldap_new_connection 1 0 0\r\n*** \u003cldap.ldapobject.SimpleLDAPObject object at 0x7fbcde13f3d0\u003e ldap://localhost:1389 - SimpleLDAPObject.set_option\r\n((17, 3), {})\r\n*** \u003cldap.ldapobject.SimpleLDAPObject object at 0x7fbcde13f3d0\u003e ldap://localhost:1389 - SimpleLDAPObject.get_option\r\n((1,), {})\r\nl's fileno=3\r\ncalling __del__ of \u003cldap.ldapobject.SimpleLDAPObject object at 0x7fbcde13f3d0\u003e\r\nldap_free_connection 1 1\r\nldap_send_unbind\r\nldap_free_connection: actually freed\r\nTraceback (most recent call last):\r\n  File \"examples/repro.py\", line 19, in \u003cmodule\u003e\r\n    simple_repro()\r\n  File \"examples/repro.py\", line 17, in simple_repro\r\n    client.close()\r\n  File \"/opt/python/python-3.11/lib64/python3.11/socket.py\", line 503, in close\r\n    self._real_close()\r\n  File \"/opt/python/python-3.11/lib64/python3.11/socket.py\", line 497, in _real_close\r\n    _ss.close(self)\r\nOSError: [Errno 9] Bad file descriptor\r\n```\r\nIf you run `strace` on it the relevant evidence is\r\n```\r\nsocketpair(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC, 0, [3, 4]) = 0\r\ngetsockname(3, {sa_family=AF_UNIX}, [128 =\u003e 2]) = 0\r\ngetsockname(4, {sa_family=AF_UNIX}, [128 =\u003e 2]) = 0\r\nwrite(1, \"client=3, server=4\\n\", 19)    = 19\r\nwrite(2, \"ldap_url_parse_ext(ldap://localh\"..., 42) = 42\r\nwrite(2, \"ldap_create\\n\", 12)           = 12\r\nwrite(2, \"ldap_url_parse_ext(ldap://localh\"..., 42) = 42\r\nwrite(2, \"ldap_new_connection 1 0 0\\n\", 26) = 26\r\nwrite(1, \"*** \u003cldap.ldapobject.SimpleLDAPO\"..., 130) = 130\r\nwrite(1, \"*** \u003cldap.ldapobject.SimpleLDAPO\"..., 127) = 127\r\nwrite(1, \"l's fileno=3\\n\", 13)          = 13\r\nwrite(1, \"calling __del__ of \u003cldap.ldapobj\"..., 79) = 79\r\nwrite(2, \"ldap_free_connection 1 1\\n\", 25) = 25\r\nwrite(2, \"ldap_send_unbind\\n\", 17)      = 17\r\nwrite(3, \"0\\5\\2\\1\\1B\\0\", 7)             = 7\r\nshutdown(3, SHUT_RDWR)                  = 0\r\nclose(3)                                = 0\r\nwrite(2, \"ldap_free_connection: actually f\"..., 37) = 37\r\nclose(4)                                = 0\r\nclose(3)                                = -1 EBADF (Bad file descriptor)\r\n```\r\n\r\nOperating system: \r\nLinux\r\nPython version: \r\nPython 3.11.8\r\npython-ldap version: \r\n3.4.0","author":{"url":"https://github.com/ClydeByrdIII","@type":"Person","name":"ClydeByrdIII"},"datePublished":"2024-08-30T20:47:06.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":1},"url":"https://github.com/575/python-ldap/issues/575"}