The Problem

We want our function to type hint such that only objects with a particular method e.g. .attack(), can be received. Something like this:

def all_attack(units: Sequence(HasAttackMethod)):

One of the objects only has a .fire() method. How do we make sure it can meet the requirement and be used in the same way as the other objects?

Extend the External Object – Make a Wrapper

Here we simply make a light wrapper around the object that doesn’t change the interface apart from adding the missing method. The new method is added to the object using setattr and everything else is delegated to the original object.

Now we can supply the extended object and it will function as required. It will also meet the required type specification, meeting the Attacker Protocol by having an .attack() method.

"""
OK - we have one weird external object - how do we handle it?
"""
from functools import partial
from typing import Protocol, Sequence


class Soldier:
    def __init__(self, name):
        self.name = name
    def attack(self):
        print(f"Soldier {self.name} swings their sword!")

class Archer:
    def __init__(self, name):
        self.name = name
    def attack(self):
        print(f"Archer {self.name} fires their arrow!")

class Catapult:
    def __init__(self, name):
        self.name = name
    def fire(self):  ## not the required name
        print(f"Catapult {self.name} hurls their fireball!")

s1 = Soldier('Tim')
s2 = Soldier('Sal')
a1 = Archer('Cal')
c1 = Catapult('Mech')


class Extend():
    def __init__(self, original, method_name, method):
        self.original = original
        ## being a method self needs to come first
        method_with_self_arg = partial(method, self=self)
        ## Add method e.g. .attack() 
        setattr(self, method_name, method_with_self_arg)
    def __getattr__(self, name, *args, **kwargs):
        """
        Delegate everything other than the added method
        to the original object
        """
        original_attr = self.original.__getattribute__(name)
        if hasattr(original_attr, '__call__'):  ## handle methods
            return original_attr(*args, **kwargs)
        else:  ## handle ordinary attributes
            return original_attr

def catapult_attack(self):
    self.original.fire()

c1_extended = Extend(original=c1, method_name='attack',
    method=catapult_attack)

class Attacker(Protocol):
    def attack(self) -> None:
        ...

def all_attack(units: Sequence[Attacker]):
    for unit in units:
        unit.attack()

all_attack([s1, s2, a1, c1_extended])
print(c1_extended.name)

Making a Polymorpher

Another approach is to make a polymorpher function. This relies upon the classes pip package https://pypi.org/project/classes/. See also: https://classes.readthedocs.io/en/latest/pages/supports.html#regular-types

As at the time of writing there seem to be problems making this code pass mypy which defeats the purpose. The explanation seems to be some brittleness in the (very promising) classes package but that is not fully confirmed.

For example, if you were making a game and you needed a function to only accept objects with an attack() method you could do it like this:

def all_attack(units: Sequence[Supports[RequiredAttackType]]):
    for unit in units:
        unit.attack()

There are some nasty and opaque steps required to make this work. These can be concentrated in one function to make it easy to use this duck hinting approach. In the code below the nastiest part is in polymorpher(). The function is included to show how it works and so others can potentially improve on it.

"""
Purpose: in a static-typing context,
how do we enable external objects
to comply with type requirements in a function?

E.g. all_attack(units: Sequence[RequiredType])
which only permits sequences of the required type.

That is no problem with Soldier and Archer
as they have attack methods.
But Catapult doesn't have an attack method.
How do we give it one without:
* changing the source code
* monkey patching

And how do we type hint (enforced by mypy) so that only objects which have the attack method can be received?

Basically it would be nice if we could add an attack method to
Catapult using a decorator e.g. like

@attack.instance(Catapult)
def _fn_catapult(catapult: Catapult):
    print(f"Catapult {catapult.name} hurls their fireball!")

And, instead of saying which types are acceptable,
follow a duck typing approach and say
what behaviours are required e.g. must have an attack method. E.g.

def all_attack(units: Sequence[Supports[RequiredAttackType]]):
    …

To make that all work, we have made a nasty, opaque function (polymorpher) that gives us both parts needed - the decorator,
and the behaviour type.
"""

from typing import Protocol, Sequence

## pip installation required
from classes import AssociatedType, Supports, typeclass

## UNITS *********************************************************

class Soldier:
    def __init__(self, name):
        self.name = name
    def attack(self):
        print(f"Soldier {self.name} swings their sword!")

class Archer:
    def __init__(self, name):
        self.name = name
    def attack(self):
        print(f"Archer {self.name} fires their arrow!")

## assume this class defined an object we can't / shouldn't modify
class Catapult:
    def __init__(self, name):
        self.name = name

s1 = Soldier('Tim')
s2 = Soldier('Sal')
a1 = Archer('Cal')
c1 = Catapult('Mech')

## ATTACKING *******************************************************
Opaque, somewhat nasty function that makes it easy to make external objects polymorphic so they'll work with a function requiring 
compliant objects

def polymorpher(*, method_name: str):
    ## want separate types depending on method_name
    RequiredType = type(method_name, (AssociatedType, ), {})
    @typeclass(RequiredType)
    def fn(instance):
        pass
    def self2pass(self):
        pass
    RequiredProtocol = type(
        f'{method_name}_protocol',
        (Protocol, ),
        {'_is_runtime_protocol': True,
             method_name: self2pass,
        })
    @fn.instance(protocol=RequiredProtocol)
    def _fn_run_method(obj: RequiredProtocol):
        return getattr(obj, method_name)()
    return fn, RequiredType

## method_name and returned fn don't have to share name
attack, RequiredAttackType = polymorpher(method_name='attack')

## now easy to make external objects polymorphic
so they'll work with a function requiring compliant objects

@attack.instance(Catapult)
def _fn_catapult(catapult: Catapult):
    print(f"Catapult {catapult.name} hurls their fireball!")

def all_attack(units: Sequence[Supports[RequiredAttackType]]):
    for unit in units:
        attack(unit)  ## note - not unit.attack()

## UNITs ATTACKing ****************************************************

units = [s1, s2, a1, c1]
all_attack(units)

Wrap Up

Type hinting and duck typing are both parts of modern Python and they need to play together well – ideally in a simpler and standardised way. The code above hopefully assists with that evolution.

See also https://www.daan.fyi/writings/python-protocols and https://medium.com/alan/python-typing-with-mypy-progressive-type-checking-on-a-large-code-base-74e13356bd3a

And a big thanks to Ben Denham for doing most of the heavy lifting with making the polymorpher (almost ;-)) work.

Type hinting and Duck Typing are both central parts of modern Python. How can we get them to work together?

Type Hinting

Type hinting allows us to indicate what types a function expects as arguments and what types it will return. E.g.

def get_greeting(name: str, age: int) -> str:

As the name makes clear, type hints are hints only but there are tools that enable type hinting to be checked and enforced.

Type hinting is still maturing in Python and more recent versions are less verbose and more readable e.g. int | float rather than typing.Union[int, float]

Initially I didn’t like type hinting. I suspected it was a costly and ritualistic safety behaviour rather than a way of writing better code. And people can certainly use type hinting like that. But I have changed my overall position on Python type hinting.

There are at least three ways to use type hinting in Python:

• To improve readability and reduce confusion. For example, what is the following function expecting for the date parameter?
  
  def myfunc(date):
  
  Is it OK if I supply date as a number (20220428) or must it be a string (“20220428”) or maybe a datetime.date object? Type hinting can remove that confusion
  
  def myfunc(date: int):
  
  It is now much more straightforward to consume this function and to modify it with confidence. I strongly recommend using type hinting like this.
  
  The following is very readable in my view:
  
  def get_data(date: int, *, show_chart=False) -> pd.DataFrame:
  
  Note: there is no need to add : bool when the meaning is obvious (unless wanting to use static checking as discussed below). It just increases the noise-to-signal ratio in the code.
  
  def get_data(date: int, *, show_chart: bool=False) -> pd.DataFrame:
  
  On a similar vein, if a parameter obviously expects an integer or a float I don’t add a type hint. For example type hinting for the parameters below reduces readability for negligible practical gain:
  
  def create_coord(x: int | float, y: int | float) -> Coord:
  
  ## knowing mypy considers int a subtype of float
  def create_coord(x: float, y: float) -> Coord
  
  Instead
  
  def create_coord(x, y) -> Coord:
  
  is probably the right balance. To some extent it is a matter of personal taste.
  
  I hope people aren’t deterred from using basic type hinting to increase readability by the detail required to fully implement type hinting.
  
• To enable static checks with the aim of preventing type-based bugs – which potentially makes sense when working on a complex code base worked on by multiple coders. Not so sure about ordinary scripts – the costs can be very high (see endless Stack Overflow questions on Type Hinting complexities and subtleties).
  
• For some people I suspect type hinting is a ritual self-soothing behaviour which functions to spin out the stressful decision-making parts of programming. Obviously I am against this especially when it makes otherwise beautiful, concise Python code “noisy” and less readable.

Duck Typing

Python follows a Duck Typing philosophy – we look for matching behaviour not specific types. If it walks like a duck and quacks like a duck it’s a duck!

For example, we might not care whether we get a tuple or a list as long as we can reference the items by index. Returning to the Duck illustration, we don’t test for the DNA of a Duck (its type) we check for behaviours we’ll rely on e.g. can it quack?

There are pros and cons to every approach to typing but I like the way Python’s typing works: strong typing (1 != ‘1’); dynamic typing (defined at run-time); and duck typing (anything as long as it quacks).

Structural Type Hinting using Protocol

If we were able to blend type hinting with duck typing we would get something where we could specify accepted types based on the behaviours they support.

Fortunately this is very easy in Python using Protocol. Below I contrast Nominal Type Hinting (based on the names of types) with Structural Type Hinting (based on internal details of types e.g. behaviours / methods)

Full Example in Code

from typing import Protocol

class AttackerClass:
    def attack(self):
       pass

class Soldier(AttackerClass):
    def __init__(self, name):
        self.name = name
    def attack(self):
        print(f"Soldier {self.name} swings their sword!")

class Archer(AttackerClass):
    def __init__(self, name):
        self.name = name
    def attack(self):
        print(f"Archer {self.name} fires their arrow!")

class Catapult:
    def __init__(self, name):
        self.name = name
    def attack(self):
        print(f"Catapult {self.name} hurls their fireball!")

## only accept instances of AttackerClass (or its subclasses)
def all_attack_by_type(units: list[AttackerClass]):
    for unit in units:
        unit.attack()

s1 = Soldier('Tim')
s2 = Soldier('Sal')
a1 = Archer('Cal')
c1 = Catapult('Mech')

all_attack_by_type([s1, s2, a1])

## will run but not wouldn't pass static check
## because c1 not an AttackerClass instance
## (or the instance of a subclass)

## comment out next line if checking with mypy etc - will fail
all_attack_by_type([s1, s2, a1, c1])

class AttackerProtocol(Protocol):
    def attack(self) -> None:
        … ## idiomatic to use ellipsis

def all_attack_duck_typed(units: list[AttackerProtocol]):
    for unit in units:
        unit.attack()

## will run as before even though c1 included
## but will also pass a static check
all_attack_duck_typed([s1, s2, a1, c1])

But what if we cannot or should not modify an object by adding the required method to it directly e.g. code in library code? That is the topic of the next blog post.

Named tuples (collections.namedtuple) are a very convenient way of gathering data together so it can be passed around. They provide a guaranteed data structure and parts can be referenced by name. It is even possible to set defaults, albeit in a clunky way. And they’re fast and lightweight (see https://death.andgravity.com/namedtuples). What more could you want!

Example:

"""
Process Points Of Interest (POIs)
“””
from collections import namedtuple
POI = namedtuple('POI', 'x, y, lbl')
pois = []
for … :
pois.append(POI(x, y, city))
… (maybe in a different module)
for poi in pois:
logging.info(f"{poi.lbl}")

.lbl will always refer to a label and I can add more fields to POI in future and this code here will still work. If I change lbl to label in POI this will break hard and obviously.

An alternative is data classes – a more readable version of standard named tuples that makes defaults much easier to use. Hmmmm – maybe these are better. They arrived in 3.7 so instead of:

from collections import namedtuple
## Note - defaults apply to rightmost first
POI = namedtuple('POI', 'x, y, lbl', defaults=['Unknown'])

We can now use:

from dataclasses import dataclass
@dataclass(frozen=True, order=True)
class POI:
x: float
y: float
lbl: str = 'Unknown'

The readable type hinting is nice too and the data structure self-documents.

Not bad but I have concluded data classes are generally an inferior version of what we can make with typing.NamedTuple. Instead of:

from dataclasses import dataclass
@dataclass(frozen=True, order=True)
class POI:
    x: float
    y: float
    lbl: str = 'Unknown'

We can use:

from typing import NamedTuple
class POI(NamedTuple):
    x: float
    y: float
    lbl: str = 'Unknown'

which has immutability and ordering out of the box not to mention having less boilerplate.

Data classes are only superior as far as I can see when mutability is wanted (frozen=False).

Another black mark against data classes – you have to import asdict from dataclasses to create a dictionary whereas namedtuples (and NamedTuples) have the _asdict() method built in.

Note – it very easy to swap from NamedTuple to dataclass should the need for mutability ever arise – all the field definitions are the same.

So the verdict is in favour of typing.NamedTuple.

That’s a lot of detail on something minor but I do use small data structures of this sort a lot to make code safer and more readable. Vanilla tuples are usually dangerous (they force you to use indexing referencing which is brittle and unreadable); and dictionaries require consistent use of keys and it is potentially painful to change keys. NamedTuples seem like a perfect balance in Python. Make them a standard part of your coding practice.