What's new in Python 3.11?
The first beta release of Python 3.11 is out, bringing some fascinating features for us to tinker with. This is what you can expect to see in 2022's release of Python later this year.
Even better error messages
Python 3.10 gave us better error messages in various regards, but Python 3.11 aims to improve them even more. Some of the most important things that are added to error messages in Python 3.11 are:
Exact error locations in tracebacks
Until now, in a traceback, the only information you got about where an exception got raised was the line. The issue could have been anywhere on the line though, so sometimes this information was not enough.
Here's an example:
This code results in an error, because one of these fields in the dictionary is None. This is what we get:
But it is impossible to tell by the traceback itself, which part of the calculation caused the error.
On 3.11 however:
It is crystal clear, that data['profits']['yearly'] was None.
To be able to render this information, the end_line and end_col data was added to Python code objects. You can also access this information directly through the obj.__code__.co_positions() method.
Notes for exceptions
To make tracebacks even more context rich, Python 3.11 allows you to add notes to exception objects, which get stored in the exceptions, and displayed when the exception is raised.
Take this code for example, where we add important information about some API data conversion logic:
This added note gets printed just below the Exception message:
Built in toml support
The standard library now has built-in support for reading TOML files, using the tomllib module:
tomllib is actually based on an open source TOML parsing library called tomli. And currently, only reading TOML files is supported. If you need to write data to a TOML file instead, consider using the tomli-w package.
asyncio Task Groups
When doing asynchronous programming, you often run into situations where you have to trigger many tasks to run concurrently, and then take some action when they are completed. For example, downloading a bunch of images in parallel, and then bundling them to a zip file at the end.
To do that, you need to collect tasks and pass them to asyncio.gather.Here's a simple example of parallelly running tasks with the gather function:
But having to maintain a list of the tasks yourself to be able to await them is a bit clunky. So now a new API is added to asyncio called Task Groups:
When the asyncio.TaskGroup() context manager exits, it ensures that all the tasks created inside it have finished running.
Bonus: Exception groups
A similar feature was also added for exception handling inside async tasks, called exception groups.
Say you have many async tasks running together, and some of them raised errors. Currently Python's exception handling system doesn't work well in this scenario.
Here's a short demo of what it looks like with 3 concurrent crashing tasks:
When you run this code:
There's no indication that 3 of these tasks were running together. As soon as the first one fails, it crashes the whole program.
But in Python 3.11, the behaviour is a bit better:
The exception now tells us that we had 3 errors thrown, in a structure known as an ExceptionGroup.
Exception handling with these exception groups is also interesting, you can either do except ExceptionGroup to catch all the exceptions in one go:
Or you can catch them based on the exception type, using the new except* syntax:
The typing module saw a lot of interesting updates this release. Here are some of the most exciting ones:
Support for variadic generics has been added to the typing module in Python 3.11.
What that means is that, now you can define generic types that can take an arbitrary number of types in them. It is useful for defining generic methods for multi-dimensional data.
Variadic generics can be really useful for defining functions that map over N-dimensional data. This feature can help a lot in type checking codebases that rely on data science libraries such as numpy or tensorflow.
The new Generic[*Shape] syntax is only supported in Python 3.11. To use this feature in Python 3.10 and below, you can use the typing.Unpack builtin instead: Generic[Unpack[Shape]].
singledispatch now supports unions
functools.singledispatch is a neat way to do function overloading in Python, based on type hints. It works by defining a generic function, and decorating it with @singledispatch. Then you can define specialized variants of that function, based on the type of the function arguments:
By inspecting the type given to the function's arguments, singledispatch can create generic functions, providing a non object-oriented way to do function overloading.
But this is all old news. What Python 3.11 brings is that now, you can pass union types for these arguments. For example, to register a function for all the number types, previously you would have to do it separately for each type, such as float, complex or Decimal:
But now, you can specify all of them in a Union:
And the code will work exactly as expected.
Previously, if you had to define a class method that returned an object of the class itself, adding types for it was a bit weird, it would look something like this:
To be able to say that a method returns the same type as the class itself, you had to define a TypeVar, and say that the method returns the same type T as the current class itself.
But with the Self type, none of that is needed:
Required and NotRequired
TypedDict is really useful to add type information to a codebase that uses dictionaries heavily to store data. Here's how you can use them:
However, TypedDicts had a limitation, where you could not have optional parameters inside a dictionary, kind of like default parameters inside function definitions.
For example, you can do this with a NamedTuple:
This was not possible with a TypedDict (at least without defining multiple of these TypedDict types). But now, you can mark any field as NotRequired, to signal that it is okay for the dictionary to not have that field:
NotRequired is most useful when most fields in your dictionary are required, having a few not required fields. But, for the opposite case, you can tell TypedDict to treat every single field as not required by default, and then use Required to mark actually required fields.
For example, this is the same as the previous code:
contextlib has a small addition to it, which is a context manager called chdir. All it does is change the current working directory to the specified directory inside the context manager, and set it back to what it was before when it exits.
One potential usecase can be to redirect where you write the logs to:
This way, you don't have to worry about changing and reverting the current directory manually, the context manager will do it for you.
In addition to all of these, there's a cherry on top: Python also got 22% faster on average in this release. It might even get faster by the time the final release is out around October!
Also, work on Python 3.12 has already started. If you want to stay up to date with all the latest developments with the language, you can check out the pull requests page on Python's GitHub repository.