René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Very slow performance with firebase_admin.auth.list_users().iterate_all()","articleBody":"  * Operating System version: n/a\r\n  * Firebase SDK version: firebase-admin==5.0.2 \r\n  * Firebase Product: auth, user management\r\n  * Python version: 3.8\r\n  * Pip version: n/a\r\n\r\n### [REQUIRED] Step 3: Describe the problem\r\n\r\nWhen fetching all user accounts using the firebase_admin.auth.list_users().iterate_all() helper, performance is very poor, taking about 5 seconds to fetch 1000 users.\r\n\r\nI have a project which has several million users. There is a scheduled task that fetches all the users for processing, but the task takes way too long and runs into problems with App Engine request timeouts.\r\n\r\n#### Steps to reproduce:\r\n\r\nHere is an App Engine standard + Python 3.8 project which has 2 routes to show the difference between iterating through all users using the iterate all helper, versus fetching each page directly:\r\n\r\nhttps://gist.github.com/davidwtbuxton/c6bbdc1e93f686cae931d1e38b5bd555\r\n\r\nThe code logs a message for every 1000 users. In my testing it takes about 5 seconds for every 1000 users on the slow path (using iterate_all) and about 1 second for every 1000 users on the fast path (not using iterate_all).\r\n\r\n\r\n#### Relevant Code:\r\n\r\n\r\nI've got a WIP patch here https://github.com/davidwtbuxton/firebase-admin-python/commit/20991aafcb007a9d4c7fc4bd8589ebff5de1ab16\r\n\r\nI need to test that WIP patch some more, but would this be useful? Suggestions for improving the patch?\r\n\r\nI think the cause of the poor performance is the [PageIterator class][1] and how next() is implemented. In the next() method, the code evaluates `self.items` 3 times. For the [_UserIterator class][2] the items list is actually a computed property, which accesses [ListUsersPage.users][3], which is also a computed property that builds a new list of user records.\r\n\r\nSo for each user in the list of users, the iterator ends up triggering a list of all users in the current page to be constructed 3 times, just to access 1 user. When a page has 1000 users, this ends up being much slower (1000 x 3) than the ideal case of constructing the list just 1 time for each page.\r\n\r\n[1]: https://github.com/firebase/firebase-admin-python/blob/01db7eb8da6094e09fc0311930718deec5ccd4ad/firebase_admin/_auth_utils.py#L35\r\n[2]: https://github.com/firebase/firebase-admin-python/blob/01db7eb8da6094e09fc0311930718deec5ccd4ad/firebase_admin/_user_mgt.py#L845\r\n[3]: https://github.com/firebase/firebase-admin-python/blob/01db7eb8da6094e09fc0311930718deec5ccd4ad/firebase_admin/_user_mgt.py#L357","author":{"url":"https://github.com/davidwtbuxton","@type":"Person","name":"davidwtbuxton"},"datePublished":"2021-09-08T11:04:46.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":0},"url":"https://github.com/571/firebase-admin-python/issues/571"}