René's URL Explorer Experiment


Title: CPython Internals (Video) – Real Python

Open Graph Title: CPython Internals – Real Python

Description: In the previous lesson, I talked about new features to the static typing system. In this lesson, I’ll give you an overview of some of the changes happening inside the CPython interpreter. Python is an interpreted language. By contrast, a compiled…

Open Graph Description: In the previous lesson, I talked about new features to the static typing system. In this lesson, I’ll give you an overview of some of the changes happening inside the CPython interpreter. Python is an interpreted language. By contrast, a compiled…

Opengraph URL: https://realpython.com/lessons/python-312-cpython-internals/

X: @realpython

direct link

Domain: realpython.com

Hey, it has json ld scripts: 
  {
    "@context": "https://schema.org",
    "@type": "VideoObject",
    "name": "CPython Internals",
    "description": "In the previous lesson, I talked about new features to the static typing system. In this lesson, I’ll give you an overview of some of the changes happening inside the CPython interpreter. Python is an interpreted language. By contrast, a compiled…",
    "thumbnailUrl": ["https://files.realpython.com/media/Python-3.12-Cool-New-Features_Watermarked.608b58320f27.jpg"],
    "uploadDate": "2023-10-03T14:00:00+00:00",
    "duration": "PT8M9S",
    
    "potentialAction": {
      "@type": "SeekToAction",
      "target": "https://realpython.com/lessons/python-312-cpython-internals/#t={seek_to_second_number}",
      "startOffset-input": "required name=seek_to_second_number"
    }
  }
authorReal Python
twitter:cardsummary_large_image
twitter:imagehttps://files.realpython.com/media/Python-3.12-Cool-New-Features_Watermarked.608b58320f27.jpg
og:imagehttps://files.realpython.com/media/Python-3.12-Cool-New-Features_Watermarked.608b58320f27.jpg
twitter:creator@realpython
og:typevideo.episode

Links:

https://realpython.com/
Start Herehttps://realpython.com/start-here/
Learn Python https://realpython.com/lessons/python-312-cpython-internals/
Python Tutorials →In-depth articles and video courseshttps://realpython.com/search?kind=article&kind=course&order=newest
Learning Paths →Guided study plans for accelerated learninghttps://realpython.com/learning-paths/
Quizzes & Exercises →Check your learning progresshttps://realpython.com/quizzes/
Browse Topics →Focus on a specific area or skill levelhttps://realpython.com/tutorials/all/
Community Chat →Learn with other Pythonistashttps://realpython.com/community/
Office Hours →Live Q&A calls with Python expertshttps://realpython.com/office-hours/
Podcast →Hear what’s new in the world of Pythonhttps://realpython.com/podcasts/rpp/
Books →Round out your knowledge and learn offlinehttps://realpython.com/products/books/
Reference →Concise definitions for common Python termshttps://realpython.com/ref/
Code Mentor →BetaPersonalized code assistance & learning toolshttps://realpython.com/mentor/
Unlock All Content →https://realpython.com/account/join/
More https://realpython.com/lessons/python-312-cpython-internals/
Learner Storieshttps://realpython.com/learner-stories/
Python Newsletterhttps://realpython.com/newsletter/
Python Job Boardhttps://www.pythonjobshq.com
Meet the Teamhttps://realpython.com/team/
Become a Contributorhttps://realpython.com/jobs/
Searchhttps://realpython.com/search
https://realpython.com/search
Joinhttps://realpython.com/account/join/
Sign‑Inhttps://realpython.com/account/login/?next=%2Flessons%2Fpython-312-cpython-internals%2F
Unlock This Lessonhttps://realpython.com/account/join/?utm_source=rp_lesson&utm_content=new-features-python-312
Unlock This Lessonhttps://realpython.com/account/join/?utm_source=rp_lesson&utm_content=new-features-python-312
https://realpython.com/courses/new-features-python-312/#team
What's New in Python 3.12https://realpython.com/courses/new-features-python-312/
Christopher Trudeauhttps://realpython.com/courses/new-features-python-312/#team
Recommended Tutorialhttps://realpython.com/python312-new-features/
Course Slides (.pdf)https://realpython.com/courses/new-features-python-312/downloads/new-features-python-312-slides/
Sample Code (.zip)https://realpython.com/courses/new-features-python-312/downloads/new-features-python-312-code/
Ask a Questionhttps://realpython.com/lessons/python-312-cpython-internals/#discussion
https://realpython.com/feedback/survey/course/new-features-python-312/liked/?from=lesson-title
https://realpython.com/feedback/survey/course/new-features-python-312/disliked/?from=lesson-title
Transcripthttps://realpython.com/lessons/python-312-cpython-internals/#transcript
Discussionhttps://realpython.com/lessons/python-312-cpython-internals/#discussion
00:00https://realpython.com/lessons/python-312-cpython-internals/#t=0.58
In the previous lesson, I talked about new features to the static typing system.https://realpython.com/lessons/python-312-cpython-internals/#t=0.58
In this lesson,https://realpython.com/lessons/python-312-cpython-internals/#t=4.9399999999999995
I’ll give you an overview of some of the changes happening inside the CPythonhttps://realpython.com/lessons/python-312-cpython-internals/#t=5.75
interpreter.https://realpython.com/lessons/python-312-cpython-internals/#t=9.63
00:11https://realpython.com/lessons/python-312-cpython-internals/#t=11.3
Python is an interpreted language. By contrast,https://realpython.com/lessons/python-312-cpython-internals/#t=11.3
a compiled language uses a compiler to translate your code into the machinehttps://realpython.com/lessons/python-312-cpython-internals/#t=15.61
language used by the computer,https://realpython.com/lessons/python-312-cpython-internals/#t=19.85
whereas an interpreted language reads a file and runs the instructions in thehttps://realpython.com/lessons/python-312-cpython-internals/#t=21.91
interpreter itself.https://realpython.com/lessons/python-312-cpython-internals/#t=26.44
00:28https://realpython.com/lessons/python-312-cpython-internals/#t=28.14
You can kind of think of it like the interpreter being a simulation of ahttps://realpython.com/lessons/python-312-cpython-internals/#t=28.14
computer running on your computer.https://realpython.com/lessons/python-312-cpython-internals/#t=31.76
One advantage of this is the same code can run on different machines becausehttps://realpython.com/lessons/python-312-cpython-internals/#t=34.3
it’s the interpreter’s responsibility to speak the machine’s language.https://realpython.com/lessons/python-312-cpython-internals/#t=37.95
00:42https://realpython.com/lessons/python-312-cpython-internals/#t=42.48
A disadvantage of this is it tends to be slower, as it adds a layer ofhttps://realpython.com/lessons/python-312-cpython-internals/#t=42.48
abstraction between the code and the machine. In all likelihood,https://realpython.com/lessons/python-312-cpython-internals/#t=46.48
the interpreter that you’re running is CPython.https://realpython.com/lessons/python-312-cpython-internals/#t=50.59
00:53https://realpython.com/lessons/python-312-cpython-internals/#t=53.22
There are others out there, though,https://realpython.com/lessons/python-312-cpython-internals/#t=53.22
but if you’re not sure which one you’re running,https://realpython.com/lessons/python-312-cpython-internals/#t=54.83
you’re probably running CPython.https://realpython.com/lessons/python-312-cpython-internals/#t=56.99
When you hear about performance improvements in a new release of Python,https://realpython.com/lessons/python-312-cpython-internals/#t=59.24
technically that is performance improvements in the CPython interpreter.https://realpython.com/lessons/python-312-cpython-internals/#t=62.88
01:08https://realpython.com/lessons/python-312-cpython-internals/#t=68.13
When you run a Python program using CPython,https://realpython.com/lessons/python-312-cpython-internals/#t=68.13
it instantiates an instance of an interpreter. As you can have more than onehttps://realpython.com/lessons/python-312-cpython-internals/#t=71.4
instance, this first one is called the main interpreter.https://realpython.com/lessons/python-312-cpython-internals/#t=76.14
01:20https://realpython.com/lessons/python-312-cpython-internals/#t=80.41
The main interpreter can spawn copies of itself,https://realpython.com/lessons/python-312-cpython-internals/#t=80.41
those copies being called subinterpreters.https://realpython.com/lessons/python-312-cpython-internals/#t=83.56
01:27https://realpython.com/lessons/python-312-cpython-internals/#t=87.77
Most of the parts of subinterpreters are independent of each other.https://realpython.com/lessons/python-312-cpython-internals/#t=87.77
The keyword in that sentence is most.https://realpython.com/lessons/python-312-cpython-internals/#t=91.47
The subinterpreter concept isn’t new.https://realpython.com/lessons/python-312-cpython-internals/#t=94.14
It’s actually been around since Python 1.5,https://realpython.com/lessons/python-312-cpython-internals/#t=96.86
but it operates below the level of the language.https://realpython.com/lessons/python-312-cpython-internals/#t=100.13
01:42https://realpython.com/lessons/python-312-cpython-internals/#t=102.97
Unless you’re writing an extension, you don’t have to be aware of it.https://realpython.com/lessons/python-312-cpython-internals/#t=102.97
01:48https://realpython.com/lessons/python-312-cpython-internals/#t=108.71
Remember when I said most? Well,https://realpython.com/lessons/python-312-cpython-internals/#t=108.71
there are things you have to be careful with when you run in parallel.https://realpython.com/lessons/python-312-cpython-internals/#t=110.59
You can’t have two different processes changing a single value at the samehttps://realpython.com/lessons/python-312-cpython-internals/#t=113.93
time. This causes consistency bugshttps://realpython.com/lessons/python-312-cpython-internals/#t=117.63
and other problems. To work around this,https://realpython.com/lessons/python-312-cpython-internals/#t=120.29
Python has the GIL. That’s short for global interpreter lock,https://realpython.com/lessons/python-312-cpython-internals/#t=122.87
and it is the bane of people trying to write parallel code.https://realpython.com/lessons/python-312-cpython-internals/#t=127.41
02:11https://realpython.com/lessons/python-312-cpython-internals/#t=131.26
There is a lot of work going on to try and shrink the GIL’s impact and/or gethttps://realpython.com/lessons/python-312-cpython-internals/#t=131.26
rid of it completely. In fact,https://realpython.com/lessons/python-312-cpython-internals/#t=135.93
there are two PEPs that I’m going to talk about that affect the structure ofhttps://realpython.com/lessons/python-312-cpython-internals/#t=138.33
subinterpreters.https://realpython.com/lessons/python-312-cpython-internals/#t=141.33
02:21https://realpython.com/lessons/python-312-cpython-internals/#t=141.96
With respect to the GIL, PEP 684 moves the GIL from being globalhttps://realpython.com/lessons/python-312-cpython-internals/#t=141.96
to the subinterpreter level, while PEP 554 adds Python-https://realpython.com/lessons/python-312-cpython-internals/#t=147.22
level access to this mechanism.https://realpython.com/lessons/python-312-cpython-internals/#t=151.97
This feature won’t actually be exposed until the next release, in Python 3.13.https://realpython.com/lessons/python-312-cpython-internals/#t=154.62
02:38https://realpython.com/lessons/python-312-cpython-internals/#t=158.95
Moving the GIL means moving almost all the global state,https://realpython.com/lessons/python-312-cpython-internals/#t=158.95
which is a whole bunch of work. It also causes some problems.https://realpython.com/lessons/python-312-cpython-internals/#t=163.66
Any existing extension code likely makes the assumption that the GIL is global,https://realpython.com/lessons/python-312-cpython-internals/#t=168.59
not part of the subinterpreter.https://realpython.com/lessons/python-312-cpython-internals/#t=172.27
02:54https://realpython.com/lessons/python-312-cpython-internals/#t=174.73
So part of the work here is to create a path for the extensions.https://realpython.com/lessons/python-312-cpython-internals/#t=174.73
Extensions can mark themselves as being aware of this change. If they are,https://realpython.com/lessons/python-312-cpython-internals/#t=178.4
they can take advantage of it. If they aren’t,https://realpython.com/lessons/python-312-cpython-internals/#t=182.11
then the old mechanism stays in place for backward compatibility.https://realpython.com/lessons/python-312-cpython-internals/#t=184.32
03:08https://realpython.com/lessons/python-312-cpython-internals/#t=188.81
All of this is under the hood.https://realpython.com/lessons/python-312-cpython-internals/#t=188.81
It isn’t until 3.13 that this is going to be exposed to the Python level.https://realpython.com/lessons/python-312-cpython-internals/#t=190.53
In the meantime, if you aren’t an extension writer,https://realpython.com/lessons/python-312-cpython-internals/#t=195.05
you won’t notice the change at all.https://realpython.com/lessons/python-312-cpython-internals/#t=197.61
03:20https://realpython.com/lessons/python-312-cpython-internals/#t=200.65
For more details about subinterpreters and the changes connected to that inhttps://realpython.com/lessons/python-312-cpython-internals/#t=200.65
3.12, see this tutorial.https://realpython.com/lessons/python-312-cpython-internals/#t=204.59
03:29https://realpython.com/lessons/python-312-cpython-internals/#t=209.05
I really need to get a T-shirt printed up that says everything in Python is anhttps://realpython.com/lessons/python-312-cpython-internals/#t=209.05
object. I seem to say it enough. Inside the interpreter,https://realpython.com/lessons/python-312-cpython-internals/#t=212.49
those very same objects are tracked by C structures that are kind of object-like.https://realpython.com/lessons/python-312-cpython-internals/#t=216.84
03:42https://realpython.com/lessons/python-312-cpython-internals/#t=222.22
Each one of these structures contains the object’s actual data,https://realpython.com/lessons/python-312-cpython-internals/#t=222.22
as well as some metadata that goes with it.https://realpython.com/lessons/python-312-cpython-internals/#t=225.69
The metadata is there to help track when an object is being referenced andhttps://realpython.com/lessons/python-312-cpython-internals/#t=228.9
therefore whether it can be deleted. Because all objects use ahttps://realpython.com/lessons/python-312-cpython-internals/#t=232.88
similar structure,https://realpython.com/lessons/python-312-cpython-internals/#t=237.67
even those objects that can’t be changed have mutable metadata sections.https://realpython.com/lessons/python-312-cpython-internals/#t=238.78
04:04https://realpython.com/lessons/python-312-cpython-internals/#t=244.3
Some immutable objects exist for the lifetime of the interpreter,https://realpython.com/lessons/python-312-cpython-internals/#t=244.3
which means they’ll never be garbage collected, and the metadata associated withhttps://realpython.com/lessons/python-312-cpython-internals/#t=248.29
them is unnecessary overhead.https://realpython.com/lessons/python-312-cpython-internals/#t=252.2
04:16https://realpython.com/lessons/python-312-cpython-internals/#t=256.0
PEP 683 proposes a way of doing something about this.https://realpython.com/lessons/python-312-cpython-internals/#t=256.0
In addition to having immutable objects,https://realpython.com/lessons/python-312-cpython-internals/#t=260.32
the interpreter will also have immortal objects.https://realpython.com/lessons/python-312-cpython-internals/#t=262.28
Those that are immortal don’t need the extra metadata, and it can be optimizedhttps://realpython.com/lessons/python-312-cpython-internals/#t=265.47
away. There are more immutable objects than you might think,https://realpython.com/lessons/python-312-cpython-internals/#t=269.73
and some things that will be able to become immortal include the None object,https://realpython.com/lessons/python-312-cpython-internals/#t=273.91
certain integers, and some strings.https://realpython.com/lessons/python-312-cpython-internals/#t=278.96
04:42https://realpython.com/lessons/python-312-cpython-internals/#t=282.86
These things are actually used so much that marking them as immortal canhttps://realpython.com/lessons/python-312-cpython-internals/#t=282.86
actually make a big difference. They don’t change,https://realpython.com/lessons/python-312-cpython-internals/#t=286.97
so they don’t need cache-handling code.https://realpython.com/lessons/python-312-cpython-internals/#t=289.97
04:52https://realpython.com/lessons/python-312-cpython-internals/#t=292.28
They don’t need to be synchronized across multiprocess instances,https://realpython.com/lessons/python-312-cpython-internals/#t=292.28
and by getting rid of some of the metadata, memory can be saved.https://realpython.com/lessons/python-312-cpython-internals/#t=296.03
This mechanism is purely internal to CPython. It won’t affect your code.https://realpython.com/lessons/python-312-cpython-internals/#t=300.63
05:05https://realpython.com/lessons/python-312-cpython-internals/#t=305.34
The PEP was brought by the folks at Instagram, and they have seen a significanthttps://realpython.com/lessons/python-312-cpython-internals/#t=305.34
improvement in memory usage and startup time in some of their large Djangohttps://realpython.com/lessons/python-312-cpython-internals/#t=309.32
clusters by introducing immortal objects.https://realpython.com/lessons/python-312-cpython-internals/#t=312.88
05:17https://realpython.com/lessons/python-312-cpython-internals/#t=317.75
You’re probably familiar with list, dictionary, and set comprehensions in Python.https://realpython.com/lessons/python-312-cpython-internals/#t=317.75
This is an example of a list comprehension,https://realpython.com/lessons/python-312-cpython-internals/#t=322.48
which iterates over the numbers list and creates a new list which contains thehttps://realpython.com/lessons/python-312-cpython-internals/#t=324.98
squares of the values in numbers. Generally speaking,https://realpython.com/lessons/python-312-cpython-internals/#t=329.77
comprehensions tend to be faster than their straight Python equivalent,https://realpython.com/lessons/python-312-cpython-internals/#t=333.78
and this has to do with how the interpreter can optimize them. Internally,https://realpython.com/lessons/python-312-cpython-internals/#t=337.62
these comprehensions get turned into a nested function. Yeah, well,https://realpython.com/lessons/python-312-cpython-internals/#t=342.36
and it turns out functions have overhead and can be expensive.https://realpython.com/lessons/python-312-cpython-internals/#t=346.71
05:50https://realpython.com/lessons/python-312-cpython-internals/#t=350.13
So, PEP 709 changes how comprehensions are represented internally,https://realpython.com/lessons/python-312-cpython-internals/#t=350.13
making them inline code instead of nested functions.https://realpython.com/lessons/python-312-cpython-internals/#t=355.49
The contents of a comprehension are their own little namespace.https://realpython.com/lessons/python-312-cpython-internals/#t=359.79
06:04https://realpython.com/lessons/python-312-cpython-internals/#t=364.09
Consider the n inside of my example there. It isn’t actually in the local stack.https://realpython.com/lessons/python-312-cpython-internals/#t=364.09
This is why they were originally created using a nested function, because thehttps://realpython.com/lessons/python-312-cpython-internals/#t=369.2
internal nested functions namespacing could be used for scope. To change them tohttps://realpython.com/lessons/python-312-cpython-internals/#t=373.57
inline code, a bit of wizardry is required to properly deal with the variables,https://realpython.com/lessons/python-312-cpython-internals/#t=378.49
putting them on the stack before the comprehension and removing them just after,https://realpython.com/lessons/python-312-cpython-internals/#t=382.6
essentially mimicking this part of a function’s role,https://realpython.com/lessons/python-312-cpython-internals/#t=387.2
but removing the need to do a jump call.https://realpython.com/lessons/python-312-cpython-internals/#t=390.15
06:33https://realpython.com/lessons/python-312-cpython-internals/#t=393.44
This change is doubling the performance of comprehensions. Of course,https://realpython.com/lessons/python-312-cpython-internals/#t=393.44
that’s just the comprehension part, not your entire script,https://realpython.com/lessons/python-312-cpython-internals/#t=397.31
but if you have some heavy comps or you use them a lot,https://realpython.com/lessons/python-312-cpython-internals/#t=400.43
this is free speedup for your code.https://realpython.com/lessons/python-312-cpython-internals/#t=403.16
06:47https://realpython.com/lessons/python-312-cpython-internals/#t=407.54
The last change I’m going to talk about here is a Linux-specific feature.https://realpython.com/lessons/python-312-cpython-internals/#t=407.54
Linux comes with a tool called perf, which is a profiler.https://realpython.com/lessons/python-312-cpython-internals/#t=411.44
It tracks most hardware events, as well as some software events in the OS.https://realpython.com/lessons/python-312-cpython-internals/#t=415.46
07:00https://realpython.com/lessons/python-312-cpython-internals/#t=420.81
With it,https://realpython.com/lessons/python-312-cpython-internals/#t=420.81
you can build call graphs, and there are a large number of tools out there thathttps://realpython.com/lessons/python-312-cpython-internals/#t=421.32
add additional functionality on top of perf’s output. Prior to 3.12,https://realpython.com/lessons/python-312-cpython-internals/#t=425.16
if you run perf on a Python program, you won’t see anything about Python,https://realpython.com/lessons/python-312-cpython-internals/#t=429.88
just the entry point to the interpreter and any underlying C code that getshttps://realpython.com/lessons/python-312-cpython-internals/#t=434.61
called.https://realpython.com/lessons/python-312-cpython-internals/#t=438.82
07:21https://realpython.com/lessons/python-312-cpython-internals/#t=441.2
Python 3.12 has added hooks to interact with the perf profiler.https://realpython.com/lessons/python-312-cpython-internals/#t=441.2
Doing this means Python calls can now be monitored as if they were native calls,https://realpython.com/lessons/python-312-cpython-internals/#t=446.5
and it makes it easier for you to do profiling throughout the entire code stack.https://realpython.com/lessons/python-312-cpython-internals/#t=451.07
07:36https://realpython.com/lessons/python-312-cpython-internals/#t=456.35
One advantage of this is it can help you see where the GIL is getting in yourhttps://realpython.com/lessons/python-312-cpython-internals/#t=456.35
way, at least while it’s still there. Core devs are busy trying to get rid of it.https://realpython.com/lessons/python-312-cpython-internals/#t=459.77
07:44https://realpython.com/lessons/python-312-cpython-internals/#t=464.7
As I mentioned,https://realpython.com/lessons/python-312-cpython-internals/#t=464.7
this is a Linux-only feature, and it isn’t enabled by default.https://realpython.com/lessons/python-312-cpython-internals/#t=465.51
You have to set a shell variable to make it go.https://realpython.com/lessons/python-312-cpython-internals/#t=469.94
If you’re on Linux, and you want to learn more about this feature,https://realpython.com/lessons/python-312-cpython-internals/#t=472.86
this article has a deep dive for you.https://realpython.com/lessons/python-312-cpython-internals/#t=476.34
08:00https://realpython.com/lessons/python-312-cpython-internals/#t=480.81
That’s the key parts of Python 3.12. Last up,https://realpython.com/lessons/python-312-cpython-internals/#t=480.81
I’ll summarize the course and point you at some sources of furtherhttps://realpython.com/lessons/python-312-cpython-internals/#t=483.78
investigation.https://realpython.com/lessons/python-312-cpython-internals/#t=487.27
Become a Memberhttps://realpython.com/account/join/
https://realpython.com/lessons/python-312-typing-improvements/
Overviewhttps://realpython.com/courses/new-features-python-312/
https://realpython.com/lessons/new-features-python-312-summary/
What's New in Python 3.12 (Overview) 01:07 https://realpython.com/videos/new-features-python-312-overview/
More Powerful F-Strings 06:53 https://realpython.com/videos/python-312-f-strings/
Improved Error Messages 05:22 https://realpython.com/lessons/python-312-error-messages/
The itertools, pathlib, and calendar Modules 09:31 https://realpython.com/lessons/python-312-itertools-pathlib-calendar/
Typing Improvements 03:19 https://realpython.com/lessons/python-312-typing-improvements/
CPython Internals 08:09 https://realpython.com/lessons/python-312-cpython-internals/
What's New in Python 3.12 (Summary) 02:50 https://realpython.com/lessons/new-features-python-312-summary/
Privacy Policyhttps://realpython.com/privacy-policy/

Viewport: width=device-width, initial-scale=1, shrink-to-fit=no, viewport-fit=cover

Robots: max-image-preview:large

URLs of crawlers that visited me.