René's URL Explorer Experiment

{"@context":"https://schema.org","@type":"DiscussionForumPosting","headline":"Still seeing the issue for endpoints staying out of sync","articleBody":"### What happened?\n\nThis issue https://github.com/kubernetes/kubernetes/issues/125638 was supposed to have fixed the issue where endpoint stay out of sync \r\n```\r\nI0807 14:01:51.613700       2 endpoints_controller.go:348] \"Error syncing endpoints, retrying\" service=\"test1/test-qa\" err=\"endpoints informer cache is out of date, resource version 10168236546 already processed for endpoints test1/test-qa\"\r\nI0807 14:01:51.624576       2 endpoints_controller.go:348] \"Error syncing endpoints, retrying\" service=\"test1/test-qa\" err=\"endpoints informer cache is out of date, resource version 10168236546 already processed for endpoints test1/test-qa\"\r\nI0807 14:01:51.645704       2 endpoints_controller.go:348] \"Error syncing endpoints, retrying\" service=\"test1/test-qa\" err=\"endpoints informer cache is out of date, resource version 10168236546 already processed for endpoints test1/test-qa\"\r\nI0807 14:01:51.686942       2 endpoints_controller.go:348] \"Error syncing endpoints, retrying\" service=\"test1/test-qa\" err=\"endpoints informer cache is out of date, resource version 10168236546 already processed for endpoints test1/test-qa\"\r\nI0807 14:01:51.768648       2 endpoints_controller.go:348] \"Error syncing endpoints, retrying\" service=\"test1/test-qa\" err=\"endpoints informer cache is out of date, resource version 10168236546 already processed for endpoints test1/test-qa\"\r\nI0807 14:01:51.808043       2 endpoints_controller.go:348] \"Error syncing endpoints, retrying\" service=\"test1/test2-qa\" err=\"endpoints informer cache is out of date, resource version 10168250766 already processed for endpoints test1/test2-qa\"\r\nI0807 14:01:51.930345       2 endpoints_controller.go:348] \"Error syncing endpoints, retrying\" service=\"test1/test-qa\" err=\"endpoints informer cache is out of date, resource version 10168236546 already processed for endpoints test1/test-qa\"\r\n```\r\nI also wrote a small script which would get me the out of sync endpoints compared to the endpointslices \r\n```\r\nfrom kubernetes.client import CoreV1Api, DiscoveryV1Api\r\nfrom hubspot_kube_utils.client import build_kube_client\r\nimport json\r\nimport os\r\nfrom datetime import datetime\r\n\r\ndef extract_ips_from_endpoint(endpoint):\r\n    ips = set()\r\n    if endpoint.subsets:\r\n        for subset in endpoint.subsets:\r\n            if subset.addresses:\r\n                ips.update(addr.ip for addr in subset.addresses)\r\n            if subset.not_ready_addresses:\r\n                ips.update(addr.ip for addr in subset.not_ready_addresses)\r\n    return ips\r\n\r\ndef extract_ips_from_endpoint_slice(slice):\r\n    if not slice.endpoints:\r\n        return set()\r\n    return set(address for endpoint in slice.endpoints\r\n               for address in (endpoint.addresses or []))\r\n\r\ndef compare_endpoints_and_slices(core_client, discovery_client):\r\n    all_mismatches = []\r\n\r\n    try:\r\n        namespaces = core_client.list_namespace()\r\n    except Exception as e:\r\n        print(f\"Error listing namespaces: {e}\")\r\n        return all_mismatches\r\n\r\n    for ns in namespaces.items:\r\n        namespace = ns.metadata.name\r\n        print(f\"Processing namespace: {namespace}\")\r\n\r\n        try:\r\n            endpoints = core_client.list_namespaced_endpoints(namespace)\r\n        except Exception as e:\r\n            print(f\"Error listing endpoints in namespace {namespace}: {e}\")\r\n            continue\r\n\r\n        for endpoint in endpoints.items:\r\n            name = endpoint.metadata.name\r\n\r\n            try:\r\n                slices = discovery_client.list_namespaced_endpoint_slice(namespace, label_selector=f\"kubernetes.io/service-name={name}\")\r\n            except Exception as e:\r\n                print(f\"Error listing endpoint slices for service {name} in namespace {namespace}: {e}\")\r\n                continue\r\n\r\n            endpoint_ips = extract_ips_from_endpoint(endpoint)\r\n            slice_ips = set()\r\n\r\n            for slice in slices.items:\r\n                slice_ips.update(extract_ips_from_endpoint_slice(slice))\r\n\r\n            if endpoint_ips != slice_ips:\r\n                mismatch = {\r\n                    \"namespace\": namespace,\r\n                    \"service_name\": name,\r\n                    \"endpoint_ips\": list(endpoint_ips),\r\n                    \"slice_ips\": list(slice_ips),\r\n                    \"missing_in_endpoint\": list(slice_ips - endpoint_ips),\r\n                    \"missing_in_slice\": list(endpoint_ips - slice_ips)\r\n                }\r\n                all_mismatches.append(mismatch)\r\n\r\n        print(f\"Completed processing namespace: {namespace}\")\r\n        print(\"---\")\r\n\r\n    return all_mismatches\r\n\r\ndef save_to_json(data, cluster_name):\r\n    timestamp = datetime.now().strftime(\"%Y%m%d_%H%M%S\")\r\n    filename = f\"{cluster_name}_mismatches_{timestamp}.json\"\r\n\r\n    with open(filename, 'w') as f:\r\n        json.dump(data, f, indent=2)\r\n\r\n    print(f\"Mismatch data for cluster {cluster_name} saved to {filename}\")\r\n\r\ndef main():\r\n    clusters = [\"test\"]\r\n    all_cluster_mismatches = {}\r\n\r\n    for cluster_name in clusters:\r\n        print(f\"Processing cluster: {cluster_name}\")\r\n\r\n        try:\r\n            kube_client = build_kube_client(host=\"TEST\",\r\n                              token=\"TOKEN\")\r\n\r\n            core_client = CoreV1Api(kube_client)\r\n            discovery_client = DiscoveryV1Api(kube_client)\r\n\r\n            mismatches = compare_endpoints_and_slices(core_client, discovery_client)\r\n\r\n            all_cluster_mismatches[cluster_name] = mismatches\r\n\r\n            save_to_json(mismatches, cluster_name)\r\n\r\n            print(f\"Completed processing cluster: {cluster_name}\")\r\n            print(f\"Total mismatches found in this cluster: {len(mismatches)}\")\r\n        except Exception as e:\r\n            print(f\"Error processing cluster {cluster_name}: {e}\")\r\n\r\n\r\nif __name__ == \"__main__\":\r\n    main()\r\n```\n\n### What did you expect to happen?\n\nI expect the endpoints to eventually sync and reflect the most upto date information. \n\n### How can we reproduce it (as minimally and precisely as possible)?\n\nI have just deployed the newer patch to our cluster and that has resulted in endpoints never ending up being updated if the status goes out of sync. \n\n### Anything else we need to know?\n\n_No response_\n\n### Kubernetes version\n\nClient Version: v1.29.7\r\nKustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3\r\nServer Version: v1.29.7\r\n\n\n### Cloud provider\n\n\u003cdetails\u003e\r\n\r\n\u003c/details\u003e\r\n\n\n### OS version\n\nalmalinux-9\n\n### Install tools\n\n\u003cdetails\u003e\r\n\r\n\u003c/details\u003e\r\n\n\n### Container runtime (CRI) and version (if applicable)\n\ncri-o\n\n### Related plugins (CNI, CSI, ...) and versions (if applicable)\n\n_No response_","author":{"url":"https://github.com/kedar700","@type":"Person","name":"kedar700"},"datePublished":"2024-08-07T14:22:04.000Z","interactionStatistic":{"@type":"InteractionCounter","interactionType":"https://schema.org/CommentAction","userInteractionCount":16},"url":"https://github.com/126578/kubernetes/issues/126578"}