René's URL Explorer Experiment


Title: NumPy Performance Tips: NumPy Loop vs Vectorisation

Open Graph Title: NumPy Performance Tips: NumPy Loop vs Vectorisation

Description: Python for-loop over 50M rows caused 48-minute single-core bottleneck. Vectorised cumsum cut runtime to 3 minutes — apply the code review rule.

Open Graph Description: Python for-loop over 50M rows caused 48-minute single-core bottleneck. Vectorised cumsum cut runtime to 3 minutes — apply the code review rule.

Opengraph URL: https://thecodeforge.io/python/numpy-performance-vectorisation/

direct link

Domain: thecodeforge.io


Hey, it has json ld scripts:
{
  "@context": "https://schema.org",
  "@type": "TechArticle",
  "mainEntityOfPage": {
    "@type": "WebPage",
    "@id": "https://thecodeforge.io/python/numpy-performance-vectorisation/"
  },
  "headline": "NumPy Loop vs Vectorisation: 45-Minute Training Bottleneck",
  "description": "Python for-loop over 50M rows caused 48-minute single-core bottleneck. Vectorised cumsum cut runtime to 3 minutes — apply the code review rule.",
  "keywords": "NumPy Loop vs Vectorisation: 45-Minute Training Bottleneck, Python, Advanced, Python Libraries, Python for-loop over 50M rows caused 48-minute single-core bottleneck. Vectorised cumsum cut runtime to 3 minutes — apply the code review rule.",
  "proficiencyLevel": "Expert",
  "author": {
    "@type": "Person",
    "@id": "https://thecodeforge.io/about/#naren",
    "name": "Naren",
    "url": "https://thecodeforge.io/about/"
  },
  "publisher": {
    "@type": "Organization",
    "@id": "https://thecodeforge.io/#org",
    "name": "TheCodeForge",
    "url": "https://thecodeforge.io",
    "logo": {"@type": "ImageObject", "url": "https://thecodeforge.io/assets/og-image.png"}
  },
  "datePublished": "2026-03-16T00:00:00+00:00",
  "dateModified": "2026-10-31T00:00:00+00:00",
  "url": "https://thecodeforge.io/python/numpy-performance-vectorisation/",
  "inLanguage": "en",
  "educationalLevel": "Advanced",
  "image": "https://thecodeforge.io/assets/og-image.png",
  "isPartOf": {"@id": "https://thecodeforge.io/#website"}
}
{
  "@context": "https://schema.org",
  "@type": "BreadcrumbList",
  "itemListElement": [
    {"@type": "ListItem", "position": 1, "name": "Home", "item": "https://thecodeforge.io/"},
    {"@type": "ListItem", "position": 2, "name": "Python", "item": "https://thecodeforge.io/python/"},
    {"@type": "ListItem", "position": 3, "name": "NumPy Loop vs Vectorisation: 45-Minute Training Bottleneck", "item": "https://thecodeforge.io/python/numpy-performance-vectorisation/"}
  ]
}
{
  "@context": "https://schema.org",
  "@type": "FAQPage",
  "mainEntity": [
    {
      "@type": "Question",
      "name": "Is np.vectorize() faster than a Python loop?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "No — or so marginally that the difference is noise. The NumPy documentation states directly that np.vectorize is 'provided primarily for convenience, not for performance.' It calls your Python function once per element in a Python loop, adding thin type-coercion handling on each side. Benchmarking consistently shows np.vectorize matching or slightly underperforming a hand-written Python loop on the same operation.\n\nUse np.vectorize when you want the convenience of applying a scalar Python function to array inputs without writing the loop yourself, and when you have no performance requirement. Do not use it when you expect speedup — you will be disappointed and confused."
      }
    },
    {
      "@type": "Question",
      "name": "What is the fastest way to apply a custom function to each element of a NumPy array?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "It depends on what the function does.\n\nIf the function can be expressed using NumPy operations — arithmetic, comparisons, math functions, boolean logic — express it that way directly. This is the fastest option: the loop runs in compiled C with SIMD and there is no Python overhead per element.\n\nIf the function cannot be expressed with NumPy operations because it has sequential dependencies, use Numba: decorate with @numba.njit and it compiles to LLVM machine code on the first call. Subsequent calls run at near-C speed. This is the right answer for recurrences, state machines, and operations where output[i] depends on output[i-1].\n\nIf the function calls external Python code or libraries that Numba cannot compile, there is no way to eliminate the per-element Python overhead. In that case, focus on batching the calls to reduce the per-element fixed cost, or use multiprocessing if each element's computation is large enough to justify serialization overhead."
      }
    },
    {
      "@type": "Question",
      "name": "Can I parallelise a Python loop over a NumPy array using multiprocessing?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Technically yes, practically rarely the right answer. The serialisation cost of passing large NumPy arrays between processes — pickling, copying, unpickling — often consumes more time than the computation being parallelised. For arrays above ~10MB, the IPC overhead typically dominates.\n\nFor CPU-bound element-wise operations, vectorisation or Numba @njit is faster and simpler because the computation stays in a single process with no serialisation overhead.\n\nMultiprocessing earns its place when each element requires significant independent computation that dwarfs the serialisation cost — for example, running ML model inference or a complex simulation per element, where each call takes tens of milliseconds. In that scenario, the per-element work dominates and parallelism across processes provides real throughput improvement. Use concurrent.futures.ProcessPoolExecutor for this pattern and benchmark with actual array sizes before committing to the architecture."
      }
    },
    {
      "@type": "Question",
      "name": "How do I know if my loop can be vectorised?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Ask one question: does computing result[i] require the value of result[i-1] or any earlier output?\n\nIf the answer is no — result[i] depends only on input values, not on prior outputs — the loop can be vectorised. The computation at each position is independent of all others, which is the only structural requirement for vectorisation. Find the NumPy equivalent (broadcasting, ufunc, np.where, cumsum trick) and replace the loop.\n\nIf the answer is yes — computing result[i] requires result[i-1] — the loop has a sequential dependency and cannot be vectorised with standard NumPy. Use Numba @njit to preserve the sequential structure while compiling it to machine code.\n\nA common source of confusion: loops that read from input[i-1] but write to result[i] are NOT sequential in the relevant sense. Reading from the input array at any position is fine — the input doesn't change. The constraint is only on reading from previously computed output values."
      }
    }
  ]
}

og:typewebsite
og:imagehttps://thecodeforge.io/assets/og-image.png
og:image:width1200
og:image:height630

Links:

TheCodeForge https://thecodeforge.io/
Pythonhttps://thecodeforge.io/python/
Buy me a coffee https://ko-fi.com/thecodeforge
1NumPy Basicshttps://thecodeforge.io/python/numpy-basics/
2NumPy Arrays and Operationshttps://thecodeforge.io/python/numpy-arrays-operations/
3Pandas Basicshttps://thecodeforge.io/python/pandas-basics/
4Pandas DataFrameshttps://thecodeforge.io/python/pandas-dataframes/
5Matplotlib Basicshttps://thecodeforge.io/python/matplotlib-basics/
6Seaborn for Data Visualisationhttps://thecodeforge.io/python/seaborn-data-visualisation/
7Requests Library in Pythonhttps://thecodeforge.io/python/requests-library-python/
8Flask Web Framework Basicshttps://thecodeforge.io/python/flask-web-framework-basics/
9Django Web Framework Basicshttps://thecodeforge.io/python/django-web-framework-basics/
10SQLAlchemy Basicshttps://thecodeforge.io/python/sqlalchemy-basics/
11itertools Module in Pythonhttps://thecodeforge.io/python/itertools-python/
12collections Module in Pythonhttps://thecodeforge.io/python/collections-module-python/
13datetime Module in Pythonhttps://thecodeforge.io/python/datetime-module-python/
14regex Module in Pythonhttps://thecodeforge.io/python/regex-module-python/
15threading and multiprocessing in Pythonhttps://thecodeforge.io/python/threading-multiprocessing-python/
16FastAPI Basicshttps://thecodeforge.io/python/fastapi-basics/
17Celery for Task Queues in Pythonhttps://thecodeforge.io/python/celery-task-queues-python/
18Pytest Fixtureshttps://thecodeforge.io/python/pytest-fixtures/
19Beautiful Soup Web Scrapinghttps://thecodeforge.io/python/beautiful-soup-web-scraping/
20Selenium with Pythonhttps://thecodeforge.io/python/selenium-python/
21Pydantic for Data Validationhttps://thecodeforge.io/python/pydantic-data-validation/
22Streamlit for Data Appshttps://thecodeforge.io/python/streamlit-data-apps/
23Playwright Python — Browser Automation and Testinghttps://thecodeforge.io/python/playwright-python/
24Advanced Network Interception and Mocking in Playwright Pythonhttps://thecodeforge.io/python/playwright-python-network-mocking/
25NumPy Broadcasting — How It Actually Workshttps://thecodeforge.io/python/numpy-broadcasting/
26NumPy Indexing and Slicing — Beyond the Basicshttps://thecodeforge.io/python/numpy-indexing-slicing/
27NumPy Shape Manipulation — reshape, flatten, ravel, transposehttps://thecodeforge.io/python/numpy-shape-manipulation/
28NumPy Mathematical Functions — ufuncs, aggregations and statisticshttps://thecodeforge.io/python/numpy-mathematical-functions/
29NumPy Random Module — Generating and Controlling Random Datahttps://thecodeforge.io/python/numpy-random-module/
30NumPy Linear Algebra — dot, matmul, linalg explainedhttps://thecodeforge.io/python/numpy-linear-algebra/
31NumPy Boolean Indexing and Fancy Indexinghttps://thecodeforge.io/python/numpy-boolean-fancy-indexing/
32NumPy Performance Tips — Vectorisation vs Loopshttps://thecodeforge.io/python/numpy-performance-vectorisation/
33NumPy with Pandas — How They Work Togetherhttps://thecodeforge.io/python/numpy-with-pandas/
34NumPy dtype and Memory Layout — float32, int64 and C vs F orderhttps://thecodeforge.io/python/numpy-dtype-memory/
35NumPy loadtxt and savetxt — Reading and Writing Array Datahttps://thecodeforge.io/python/numpy-loadtxt-savetxt/
36NumPy where, select and piecewise — Conditional Array Operationshttps://thecodeforge.io/python/numpy-where-select-piecewise/
37FastAPI Path Parameters and Query Parametershttps://thecodeforge.io/python/fastapi-path-query-parameters/
38FastAPI Request Body and Pydantic Modelshttps://thecodeforge.io/python/fastapi-request-body-pydantic/
39FastAPI Response Models and Status Codeshttps://thecodeforge.io/python/fastapi-response-models-status-codes/
40FastAPI Dependency Injection — How and Why to Use Ithttps://thecodeforge.io/python/fastapi-dependency-injection/
41FastAPI Authentication — JWT and OAuth2 with Password Flowhttps://thecodeforge.io/python/fastapi-authentication-jwt/
42FastAPI Background Tasks and Async Endpointshttps://thecodeforge.io/python/fastapi-background-tasks-async/
43FastAPI Database Integration with SQLAlchemyhttps://thecodeforge.io/python/fastapi-sqlalchemy-database/
44FastAPI File Uploads and Form Datahttps://thecodeforge.io/python/fastapi-file-uploads-forms/
45FastAPI Middleware — Logging, CORS and Custom Middlewarehttps://thecodeforge.io/python/fastapi-middleware-cors/
46FastAPI Testing with pytest and TestClienthttps://thecodeforge.io/python/fastapi-testing-pytest/
47FastAPI WebSockets — Real-time Communicationhttps://thecodeforge.io/python/fastapi-websockets/
48FastAPI Deployment — Docker, Uvicorn and Gunicornhttps://thecodeforge.io/python/fastapi-deployment-docker/
49FastAPI vs Flask vs Django — When to Use Whichhttps://thecodeforge.io/python/fastapi-vs-flask-vs-django/
50FastAPI Error Handling and Custom Exception Handlershttps://thecodeforge.io/python/fastapi-error-handling/
51PySpark Tutorial: Big Data Processing with Pythonhttps://thecodeforge.io/python/pyspark-tutorial/
52FastAPI CLI — dev, run and deployhttps://thecodeforge.io/python/fastapi-cli-dev-run-deploy/
53FastAPI OpenAPI Customization — Tags, Examples and Schemahttps://thecodeforge.io/python/fastapi-openapi-customization/
54FastAPI Lifespan Events — Startup, Shutdown and Context Managershttps://thecodeforge.io/python/fastapi-lifespan-events/
55FastAPI Streaming Responses and File Responseshttps://thecodeforge.io/python/fastapi-streaming-responses/
56FastAPI Async SQLAlchemy — Alembic Migrations and Production Patternshttps://thecodeforge.io/python/fastapi-async-sqlalchemy/
57FastAPI Cloud — Official Deployment Platformhttps://thecodeforge.io/python/fastapi-cloud-deployment/
58Django OperationalError No Such Table Fixhttps://thecodeforge.io/python/django-operationalerror-no-such-table/
59Django ImproperlyConfigured Fixhttps://thecodeforge.io/python/django-improperlyconfigured/
← Back to Python track https://thecodeforge.io/python/
Homehttps://thecodeforge.io/
Pythonhttps://thecodeforge.io/python/
Follow https://www.linkedin.com/in/thecodeforge-io-a0676b3b5/
 ● Production Incidenthttps://thecodeforge.io/python/numpy-performance-vectorisation/#v2-production-incident
 🔎 Debug Guidehttps://thecodeforge.io/python/numpy-performance-vectorisation/#v2-debug-guide
 ⚙ Triage Commandshttps://thecodeforge.io/python/numpy-performance-vectorisation/#v2-triage-commands
Pythonhttps://thecodeforge.io/python/introduction-python/
NumPyhttps://thecodeforge.io/python/numpy-basics/
arrayhttps://thecodeforge.io/dsa/array-data-structure/
code reviewhttps://thecodeforge.io/cs-fundamentals/code-review-best-practices/
arrayshttps://thecodeforge.io/c-cpp/arrays-c/
operatorshttps://thecodeforge.io/python/operators-python/
conditionalshttps://thecodeforge.io/javascript/conditionals-javascript/
prefix sum arrayhttps://thecodeforge.io/dsa/prefix-sum-array/
→NumPy Broadcasting — Silent OOM That Killed 5M Profiles5M profiles OOM-killed a container because broadcasting silently inflated a 2D operation into 3D. Avoid GFG's toy examplhttps://thecodeforge.io/python/numpy-arrays-operations/
→NumPy Broadcasting — The 10x Memory Blow-UpSubtracting a 1D mean from a 2D array created a 10GB intermediate, killing the kernel. Avoid silent OOM with keepdims=Trhttps://thecodeforge.io/python/numpy-basics/
→Python Performance Optimisation — O(n*m) Loop Cost87% of runtime was a linear membership check on 200K entries. That O(n*m) pattern was costly. Profile first to avoid thihttps://thecodeforge.io/python/python-performance-optimisation/
→Pandas Row Iteration: 9x Faster Vectorization Winsiterrows on 3M rows took 6 hours; vectorized code took 47s. Learn the speed ladder from itertuples to NumPy. Speed up tohttps://thecodeforge.io/python/pandas-row-iteration-vectorization/
→NumPy Conditional Operations — The 10× Slower Pipeline TrapA factory batch job missed its 30-min SLA due to nested np.where — compare np.where, np.select, and np.piecewise to avoihttps://thecodeforge.io/python/numpy-where-select-piecewise/
Text Classification with MLIntermediatehttps://thecodeforge.io/ml-ai/text-classification-ml/
Follow https://www.linkedin.com/in/thecodeforge-io-a0676b3b5/
←PreviousNumPy Boolean Indexing and Fancy Indexinghttps://thecodeforge.io/python/numpy-boolean-fancy-indexing/
NextNumPy with Pandas — How They Work Together→https://thecodeforge.io/python/numpy-with-pandas/
LinkedIn https://www.linkedin.com/in/thecodeforge-io-a0676b3b5/
Facebook https://www.facebook.com/profile.php?id=61583020033815
YouTube https://www.youtube.com/channel/UCMar7CF2EmCF1I2tw1tNVuQ
Javahttps://thecodeforge.io/java/
Pythonhttps://thecodeforge.io/python/
C / C++https://thecodeforge.io/c-cpp/
JavaScripthttps://thecodeforge.io/javascript/
DSAhttps://thecodeforge.io/dsa/
Databasehttps://thecodeforge.io/database/
System Designhttps://thecodeforge.io/system-design/
DevOpshttps://thecodeforge.io/devops/
C# / .NEThttps://thecodeforge.io/csharp/
PHPhttps://thecodeforge.io/php/
ML / AIhttps://thecodeforge.io/ml-ai/
Interviewhttps://thecodeforge.io/interview/
CS Fundamentalshttps://thecodeforge.io/cs-fundamentals/
Rusthttps://thecodeforge.io/rust/
Toolshttps://thecodeforge.io/tools/
Cheatsheetshttps://thecodeforge.io/cheatsheets/
Mobilehttps://thecodeforge.io/mobile/
Data Analyticshttps://thecodeforge.io/data-analytics/
Securityhttps://thecodeforge.io/security/
Frontendhttps://thecodeforge.io/frontend/
Data Engineeringhttps://thecodeforge.io/data-engineering/
Testinghttps://thecodeforge.io/testing/
Cloudhttps://thecodeforge.io/cloud/
Observabilityhttps://thecodeforge.io/observability/
Web Platformhttps://thecodeforge.io/web-platform/
🧯 Errors Explainedhttps://thecodeforge.io/errors/
⚒ Blueprintshttps://thecodeforge.io/blueprints/
⌨ Practicehttps://thecodeforge.io/practice/
Interview Prephttps://thecodeforge.io/interview/
DSA Problemshttps://thecodeforge.io/dsa/
Developer Toolshttps://thecodeforge.io/tools/
Cheat Sheetshttps://thecodeforge.io/cheatsheets/
Abouthttps://thecodeforge.io/about/
Sitemaphttps://thecodeforge.io/sitemap.xml
Privacyhttps://thecodeforge.io/privacy.html

Viewport: width=device-width, initial-scale=1.0

Robots: max-image-preview:large


URLs of crawlers that visited me.