Title: Python Performance Optimisation — O(n*m) Loop Cost | TheCodeForge
Open Graph Title: Python Performance Optimisation — O(n*m) Loop Cost | TheCodeForge
Description: 87% of runtime was a linear membership check on 200K entries. That O(n*m) pattern was costly. Profile first to avoid this pattern in production.
Open Graph Description: 87% of runtime was a linear membership check on 200K entries. That O(n*m) pattern was costly. Profile first to avoid this pattern in production.
Opengraph URL: https://thecodeforge.io/python/python-performance-optimisation/
Domain: thecodeforge.io
{
"@context": "https://schema.org",
"@type": "TechArticle",
"mainEntityOfPage": {
"@type": "WebPage",
"@id": "https://thecodeforge.io/python/python-performance-optimisation/"
},
"headline": "Python Performance Optimisation — O(n*m) Loop Cost",
"description": "87% of runtime was a linear membership check on 200K entries. That O(n*m) pattern was costly. Profile first to avoid this pattern in production.",
"keywords": "Python Performance Optimisation — O(n*m) Loop Cost, Python, Advanced, Advanced Python, 87% of runtime was a linear membership check on 200K entries. That O(n*m) pattern was costly. Profile first to avoid this pattern in production.",
"proficiencyLevel": "Expert",
"author": {
"@type": "Person",
"@id": "https://thecodeforge.io/about/#naren",
"name": "Naren",
"url": "https://thecodeforge.io/about/"
},
"publisher": {
"@type": "Organization",
"@id": "https://thecodeforge.io/#org",
"name": "TheCodeForge",
"url": "https://thecodeforge.io",
"logo": {"@type": "ImageObject", "url": "https://thecodeforge.io/assets/og-image.png"}
},
"datePublished": "2026-03-06T00:00:00+00:00",
"dateModified": "2026-10-31T00:00:00+00:00",
"url": "https://thecodeforge.io/python/python-performance-optimisation/",
"inLanguage": "en",
"educationalLevel": "Advanced",
"image": "https://thecodeforge.io/assets/og-image.png",
"isPartOf": {"@id": "https://thecodeforge.io/#website"}
}
{
"@context": "https://schema.org",
"@type": "BreadcrumbList",
"itemListElement": [
{"@type": "ListItem", "position": 1, "name": "Home", "item": "https://thecodeforge.io/"},
{"@type": "ListItem", "position": 2, "name": "Python", "item": "https://thecodeforge.io/python/"},
{"@type": "ListItem", "position": 3, "name": "Python Performance Optimisation — O(n*m) Loop Cost", "item": "https://thecodeforge.io/python/python-performance-optimisation/"}
]
}
{
"@context": "https://schema.org",
"@type": "FAQPage",
"mainEntity": [
{
"@type": "Question",
"name": "What is Python Performance Optimisation in simple terms?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Python Performance Optimisation is the systematic practice of finding where your Python program is slow and applying the right fix at the right layer. The discipline starts with measurement — profiling tells you where time is actually spent, which is almost never where you expect. The fixes range from switching a list to a dict (algorithmic), to using __slots__ (memory), to replacing threads with processes (concurrency), to using NumPy (native acceleration). The order matters: fix the algorithmic layer first, the memory layer second, the concurrency model third, and the native acceleration layer last."
}
},
{
"@type": "Question",
"name": "Is Python really slow compared to other languages?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Python is slower than C, Rust, or Go for raw CPU computation — typically 10-100x slower for tight numeric loops in pure Python. But for I/O-bound work — web services, API calls, database queries — the bottleneck is network latency and database throughput, not CPU speed. Python's asyncio handles tens of thousands of concurrent connections efficiently, and the language's actual computation speed is rarely the bottleneck in those workloads.\n\nThe more useful framing: Python with NumPy for numeric computation is competitive with MATLAB and R. Python with asyncio for network I/O is competitive with Node.js. Python for data engineering pipelines, after algorithmic optimisation, runs at speeds that are economically viable at very large scale — the production incident in this article went from 14 hours to 47 minutes with a one-line fix, no language change required."
}
},
{
"@type": "Question",
"name": "When should I use NumPy vs pure Python for data processing?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Use NumPy when your data is numeric, homogeneous (all the same type), and the dataset is large enough that the vectorisation benefit exceeds the array creation overhead — roughly 10K elements is a reasonable lower bound, but always benchmark your specific case. NumPy's vectorised operations execute in optimised C on contiguous memory, providing 50-500x speedups over equivalent Python loops for simple arithmetic.\n\nFor small datasets under 1K elements, a Python loop may be faster because the fixed overhead of NumPy array creation and dispatch exceeds the loop execution time. For non-numeric data (strings, mixed types, complex objects), NumPy provides limited benefit. For tabular data, Pandas is built on NumPy and provides vectorised string and categorical operations that are substantially faster than element-wise Python loops.\n\nFor complex numeric logic that cannot be cleanly expressed as NumPy array operations, Numba's @jit decorator compiles arbitrary Python numeric functions to native machine code without requiring reformulation as array operations — it handles loops, conditionals, and recursion that NumPy cannot."
}
},
{
"@type": "Question",
"name": "How do I profile a Python application running in production without restarting it?",
"acceptedAnswer": {
"@type": "Answer",
"text": "Use py-spy. It is a sampling profiler that attaches to a running Python process by reading its stack from outside the process using OS ptrace — no code changes, no restart, no instrumentation, near-zero overhead. Run py-spy top --pid for a live htop-style view of where time is being spent. Run py-spy record --pid --output profile.svg for a flame graph you can open in a browser. Both work on Docker containers and Kubernetes pods with the appropriate permissions.\n\nFor memory profiling in production, memray can attach to running processes and record allocation traces with low overhead. tracemalloc must be started at application startup and cannot be attached later, so it is better suited for staging instrumentation than live production diagnosis.\n\nFor identifying GIL contention specifically, py-spy top --gil shows which threads are blocked waiting for the GIL, which is the signature of CPU-bound threading that needs to be moved to multiprocessing."
}
},
{
"@type": "Question",
"name": "What is the fastest way to speed up a Python web API?",
"acceptedAnswer": {
"@type": "Answer",
"text": "The answer depends entirely on where the bottleneck is — which is why the answer always starts with profiling.\n\nFor I/O-bound APIs (the most common case): use an async framework (FastAPI, Starlette), async database drivers (asyncpg for PostgreSQL, motor for MongoDB), and async HTTP clients (httpx). Connection pooling is critical — opening a new database connection per request is the most common source of unnecessary latency. Profile with py-spy on the running process to confirm the bottleneck is I/O before making framework changes.\n\nFor CPU-bound endpoints: offload to a process pool with run_in_executor(ProcessPoolExecutor) so the event loop remains free to handle other requests during the computation. For endpoints that are genuinely CPU-heavy on every request, a task queue (Celery, Dramatiq, ARQ) decouples the response from the work — return a job ID immediately, let the client poll or webhook for the result.\n\nFor both: add response caching at the HTTP layer (Redis via fastapi-cache) for responses that are identical across requests within a time window. The fastest code is code that does not execute."
}
}
]
}
| og:type | website |
| og:image | https://thecodeforge.io/assets/og-image.png |
| og:image:width | 1200 |
| og:image:height | 630 |
Links:
Viewport: width=device-width, initial-scale=1.0
Robots: max-image-preview:large