1.2 Python Type Hierarchy

apple

1.2.a. Numbers & Collections

1.2.b. Callables

• User Defined Function
• Generators - callable that are executed and only give value when called
• Class
• Instance Methods
• Class Instance (which is made using the call method)
• Builtin functions (len(), open()…)
• Builtin Methods (list.append())

1.2.c. Singletons

These are only initiated once

• None
• NotImplemented
• Ellipsis (…) operator

1.4 Variable Names

https://www.python.org/dev/peps/pep-0008/

1.4.a Conventions

• _my_var
  • This generally indicates ==“internal use”== or ==“private”== object
  • Objects named this way will not get imported by statements like from module import *
• __my_var
  • Used to mangle class attributes - used in inheritance chains
  • when a var __my_var is defined in a class, python automatically changes it to _ClassName__my_var at runtime in the class dict.
  • This __my_var persists across inheritance chains
• __my_var__
  • Used for system defined names - have special meanings

1.6 Functions

• In python def assigns a function a name
• lambda doesn’t give a name to the function but just defines a function

# def defined function
def new_function():
    pass

# lambda
lambda x: x**2

1.7 Loops

Python for loops use iterables to return values

for x in range(3): # range(3) is an iterable object which is capable of returning objects one at a time
     print(x)

outer 0x100fc91a0
inner 0x100fc91a0
the value is:  hello
(<cell at 0x101e97970: str object at 0x100fc91a0>,)
('x',)
outer 0x100fc91a0
inner 0x100fc91a0
the value is:  hello
(<cell at 0x101e979d0: str object at 0x100fc91a0>,)
('x',)

• Other data-structures like ==List==, ==Tuples==, ==Strings==, ==Dictionaries== are also iterables in python.
• But iterables and collections are different things.

1.7.a Try-Except-Finally gotcha in loops

for i in range(5):
    print("--------------------------------")
    try:
        10/(i-2)
    except ZeroDivisionError:
        print(f"Divide by zero error! while i={i}")

        '''
        this "continue" doesn't imediately exit the current iteration

        first it finishes the resolution of the finally block
        and only then it goes to the next iteration
        '''
        continue
    finally:
        print("keep on running!")

    print(i, "- running main loop")

--------------------------------
keep on running!
0 - running main loop
--------------------------------
keep on running!
1 - running main loop
--------------------------------
Divide by zero error! while i=2
keep on running!
--------------------------------
keep on running!
3 - running main loop
--------------------------------
keep on running!
4 - running main loop

1.7.b Using Iterables which contains indexes

s = "hello"

for i in range(len(s)):
    print(i, s[i])

for i, char in enumerate(s):
    print(i, char)

0 h
1 e
2 l
3 l
4 o
0 h
1 e
2 l
3 l
4 o

1.7.c Using Else with While loop

Some points to keep in mind:

1. While loop can use while True: if you are uncertain if the loop will run even a single time.
2. While loop can use the else: after the loop to check if the loop ended normally without a use of break
3. Use of break, continue is allowed inside the try-except-finally stmt but the finally of the try-block is always guranteed to run before breaking or continuing the loop

i = 1
while i > 0:
    print("--------------------------------")
    try:
        10/(i-2)
    except ZeroDivisionError:
        print(f"Divide by zero error! while i={i}")

        '''
        this "continue" doesn't imediately exit the current iteration

        first it finishes the resolution of the finally block
        and only then it goes to the next iteration
        '''
        break
    finally:
        print("keep on running!")

    print(i, "- running main loop")
    i -= 1
else:
    print("\nThe loop exectued without ZeroDivisionError!")

--------------------------------
keep on running!
1 - running main loop

The loop exectued without ZeroDivisionError!

1.9 Classes in python

class Vector:
    def __init__(self, x, y):
        self.x = x
        self.y = y

    @property
    def x(self):
        return self._x

    @x.setter
    def x(self, x):
        self._x = x

    @property
    def y(self):
        return self._y

    @y.setter
    def y(self, y):
        self._y = y

    def __repr__(self):
        return "Vector({0}, {1})".format(self.x,self.y)

    def __str__(self):
        return "Vector(x:{0}, y:{1})".format(self.x,self.y)

    def __call__(self):
        print("calling the Vector Class")

    def __eq__(self, vect):
        if isinstance(vect, Vector):
            return vect.x == self.x and vect.y == self.y


v = Vector(10,20)
print(v)

Vector(x:10, y:20)

2. Variables

2.1 Variables and Memory References

• In python id() function can be used to find the memory address referenced by the variable
• id returns a base 10 number - can be converted to hex using the hex() func

my_var = 10
print(my_var, id(my_var), hex(id(my_var)))

greetings = "hello"
print(greetings, id(greetings), hex(id(greetings)))

10 4378864416 0x105002b20
4366078320 0x1043d1170

2.2 Reference Counting

2.2.a Mechanism of ref counting

2.2.b Finding the Reference Count

• sys.getrefcount(my_var)
  • passing my_var to getrefcount() itself creates a new extra reference! - which increases the count by 1.
• ctypes.c_long.from_address(address).value
  • here we directly pass the address (an integer), not a reference- which doesn’t affect the value of reference count.

import sys

a = [1,2,3]
print(id(a), sys.getrefcount(a))

import ctypes

def ref_count(addr: int):
    return ctypes.c_long.from_address(addr).value
print(id(a), ref_count(id(a)))

b = a
print(id(b), ref_count(id(b)))

c = a
print(id(c), ref_count(id(c)))

4311562240 2
4311562240 1
4311562240 2
4311562240 3

• When the ==ref count hits 0==, python memory manager frees up the memory to be used later

2.3 Circular References & Garbage Collection

2.3.a case 1 (regular references)

• my_var points to ObjectA
• var_1(an instance variable of ObjectA) points to ObjectB

• When we delete my_var, ObjectA’s ref count becomes 0 - thus ObjectA is cleared.
• Now that ObjectA is cleared, var_1 which pointed to ObjectB also gets cleared - which makes the reference count of ObjectB also 0. so it gets cleared also

2.3.b case 2 (circular references)

• my_var points to ObjectA
• var_1(an instance variable of ObjectA) points to ObjectB
• var_2(an instance variable of ObjectB) points to ObjectA

• When we delete my_var, ObjectA’s ref count becomes 1 - thus ObjectA persists.
• Now that ObjectA is not cleared, var_1 which pointed to ObjectB also doesn’t get cleared - which makes the reference count of ObjectB stay at 1.
• So, both of these objects persist and cause memory leak.

2.3.c Garbage collection

• It plays a particularly important role in circular references scenario.
• can be controlled by using the gc module

import ctypes
import gc

def ref_count(addr: int):
    return ctypes.c_long.from_address(addr).value

def object_exists(obj_id: int):
    for obj in gc.get_objects():
        if id(obj) == obj_id:
            return True
    return False

class A:
    def __init__(self):
        # create a new instance of B and pass self(A's current instance) as the constructor arg
        self.b = B(self)
        print(f"A: self: { hex(id(self)) }, b: { hex(id(self.b)) }")

class B:
    def __init__(self, a):
        self.a = a
        print(f"B: self: { hex(id(self)) }, a: { hex(id(self.a)) }")

# disable gc as to not automatically the circular reference
gc.disable()

my_var = A()

a_id = id(my_var)
b_id = id(my_var.b)

print("a", ref_count(a_id))
print("b", ref_count(b_id))

print(object_exists(a_id), object_exists(b_id))

my_var = None

print("a", ref_count(a_id))
print("b", ref_count(b_id))

gc.collect()

print(object_exists(a_id), object_exists(b_id))

print("a", ref_count(a_id))
print("b", ref_count(b_id))

B: self: 0x1002cb230, a: 0x1002cbe00
A: self: 0x1002cbe00, b: 0x1002cb230
a 2
b 1
True True
a 1
b 1
False False
a 0
b 0

2.4 Variable Reassignment

2.4.a Example 1 for an int

• my_var contains a reference to the Object at 0x1000
• when my_var is reassigned, the object at 0x1000 doesn’t change - rather a new object 15 is created at 0x1234 and the reference to that object is now stored at my_var

2.4.b Example 2 for an int

• when my_var contains a reference to the Object at 0x1000
• when my_var is reassigned, the result of computation (my_var + 5) is stored in a completely different address and similar to before the reference to that object is then stored in my_var.

This is particularly important in Objects like integers whose internal states are never modified. This can be studied further in [[Mutability of Objects in Python]].

a = 10
print(hex(id(a)))

a = 15
print(hex(id(a)))

a = a + 5
print(hex(id(a)))

2.5 Object Mutability

• Objects like classes instances are mutable - their internal state can be changed

2.5.a More Examples

# container(tuple) and elements(integers) both immutable
# tuple contains refrence to immutable objects(int)
tup = (1,2,3)

# container(tuple) immutable but elements(lists) mutable
# tuple contains refrence to mutable objects(list)
a = [1,2]
b = [2,3]
c = [3,4]

a_id = id(a)

t = (a,b,c)

a.append(10)
b.append(20)
c.append(30)

print(t)
print( hex(id(a)), hex(id(t[0])), hex(a_id))

# address of 1 at tup[0] & 1 at a[0] is the same
print(id(tup[0]), id(a[0])) # an example of integer interning

([1, 2, 10], [2, 3, 20], [3, 4, 30])
0x1022391c0 0x1022391c0 0x1022391c0
4345260544 4345260544

2.5.b Appending & Concatenating are different kinds of operations

# appending and concatinating are inherently different kinds of operation
# when concatinating, LHS is evaluated then RHS is evaluated, then combined then the resut is returned as a new Object, which will be at a new mem address
a = [1,2,3]
a_id = id(a)

a.append(4)
print(hex(id(a)), hex(a_id))

a = a + [5]
print(hex(id(a)), hex(a_id))

2.6 Function args and Mutability

2.6.a How immutable objects are safe from unintended side-effects

• Strings (str) in python are immutable objects

2.6.b How mutable objects are not safe from unintended side-effects

def process_immut(s: str):
    s = s + " world"
    return s

my_str = "hello"
new_str = process_immut(my_str)

print("original unmodified in immutable", id(my_str), id(new_str))

def process_mut(lst: list):
    lst.append(10)
    return lst

my_lst = [1,2,3]
new_lst = process_mut(my_lst)

print("original modified in mutable", id(my_lst), id(new_lst))

4369830304 4369805680
4369802304 4369802304

2.6.b Examples based on tuples

tup = ([0,2],1)
tup_id = id(tup)


# The references to even the mutable items of a tuple cannot me modified
try:
    tup[0] = tup[0] + [4]
except TypeError:
    print("Items of a tuple cannot be modified")


# The internal states of a tuple element(which is mutable) can be modified
tup[0].append(3)


print(tup_id, id(tup))

Items of a tuple cannot be modified
4307823232 4307823232

2.7 Shared references and Mutability

• Shared references - when two or more variables have the reference to the same object

2.7.a Some examples demonstrating shared references

a = 10
b = a
print("a and b have a shared reference: ", a is b)


y = "value"
def modify(v):
    print("y and v have a shared reference: ", y is v)
    pass
modify(y)


# But there are caviats to this, check integer interning in python to find out more!
c = 10
d = 10
print("c and d have a shared reference: ", c is d)

# But there are caviats to this, check string interning in python to find out more!
i = "hello"
j = "hello"
print("i and j have a shared reference: ", i is j)


lst_1 = [1,2,3]
lst_2 = lst_1
print("lst_1 and lst_2 have a shared reference: ", lst_1 is lst_2)
lst_2.append(4)
print("lst_1 and lst_2 after append have a shared reference: ", lst_1 is lst_2)
lst_1 = lst_2 + [5] # in this case the result of concat is stored as a new object in new address
print("lst_1 and lst_2 after concat have a shared reference: ", lst_1 is lst_2)


lst_3 = [1,2,3]
lst_4 = [1,2,3]
print("lst_3 and lst_4 have a shared reference: ", lst_1 is lst_2)

a and b have a shared reference:  True
y and v have a shared reference:  True
c and d have a shared reference:  True
i and j have a shared reference:  True
lst_1 and lst_2 have a shared reference:  True
lst_1 and lst_2 after append have a shared reference:  True
lst_1 and lst_2 after append have a shared reference:  False
lst_3 and lst_4 have a shared reference:  False

2.8 Variable Equality

Two fundamental ways to compare a variable:

• using is - identity operator
• using == - equality operator

2.8.a “is” operator (Memory Address)

• compares memory address
• essentially does id(a) == id(b)
• not can be written as not(a is b) or a is not b

2.8.a ”\==” operator (Internal State/data)

• compares the internal state/data
• not can be written as a != b or not(a == b)

2.8.c Examples and Differences of “is” and ”\==” operators

2.8.d None object

• None is not nothing - None is a real object managed by the Python Memory Manager.
• All the None mentioned in the code, always has the shared reference to the same None singleton.

a = None
b = None
c = None
print(id(a) == id(b) == id(c))

True

2.9 Everything is an object in python

• Data types

  • Integers (int)
  • Booleans (bool)
  • Floats (float)
  • Strings (str)
  • Lists (list)
  • Tuples (tuple)
  • Sets (set)
  • Dictionaries (dict)
  • None (NoneType)

• Other constructs

  • Operators (+, -, \==, is, …)
  • Functions (function)
  • Classes (class)
  • Types (type)

• All of the above are all Objects - they are all instances of Classes

• All of them have memory address

2.10 Memory Optimizations

2.10.a Python Interning - reusing objects on demand

2.10.a.i Integer Interning

• CPython at startup pre-loads(caches) a global list of integer objects(-5 to 256)
• Any time the code references an integer in that range the interned object is used.
• So, integer objects in range (-5 to 256) are Singletons

c = 252
d = 252
print(id(c) == id(d)) # True

a = -10
b = -10
print(id(a) == id(b)) # False

i = 10
j = int(10)
k = int('10')
l = int('1010' , base=2)

print(id(i), id(j), id(k), id(l)) # all of them have the same address

True
True
4394789664 4394789664 4394789664 4394789664

2.10.a.i String Interning

• Not all strings are interned
• As python code is compiled, certain strings are interned: - identifiers
  • variable names
  • function name
  • class name
  • etc
• Strings that look like identifiers are interned - but not guaranteed
• This is done to optimize speed and or memory
• Strings can also be forced to be interned using the sys.intern()

import sys

a = sys.intern("apple ball cat")
b = sys.intern("apple ball cat")
print(id(a), id(b))

a = "apple"
b = "ball"
print(id(a), id(b)) # Maybe interned

2.10.b Peephole Optimizations

• compile time optimization

2.10.b.i Constant Expression Optimizations

• Expressions like 24 * 60 will be set to 1440 at comp time
• Short sequences - length less than 20
  • (1 , 2) * 5 will be set to (1,2,1,2,1,2,1,2,1,2)
  • “abc” * 5 will be set to “abcabcabcabcabc”
• If length is more than 20, then python decides it is not worth the tradeoff -“the quick brown fox” * 100 is not optimized

2.10.b.ii Membership Tests: Mutable Objects are replaced By Immutable Objects

• constant mutable expressions are replaced by their immutable counterpart
• list convert to tuple
• set convert to frozen_set
• Examples:
  • if i in [1,2,3]: is replaced by if i in (1,2,3)
• Because Set are much faster than List for membership tests
  • if i in [1,2,3]: can be written as if i in {1,2,3} - where {1,2,3} is a set

def my_func():
 a = 24 * 60
 b = (1, 2) * 5
 c = 'abc' * 3
 e = "apple ball cat dog elephant" * 5
 f = ["a", "b", "c"] * 4

 my_func.__code__.co_consts

 '''
 Returns

 (None, 1440, (1, 2, 1, 2, 1, 2, 1, 2, 1, 2), 'abcabcabc', 'apple ball cat dog elephantapple ball cat dog elephantapple ball cat dog elephantapple ball cat dog elephantapple ball cat dog elephant', ('a', 'b', 'c'), 4)
 '''

3. Numerical Datatypes in Python

3.1 Introduction to numerical types

4. Functions

4.1 Unpacking Iterables

4.1.a Side Note on Tuples

• Tuples are not defined by (), but rather by ,
• () are just used to make tuples clearer

4.1.b Packed Values (Iterables)

• Any values that are bundled are packed values
• Examples:
  • Tuples & Lists
  • String
  • Sets & Dictionaries

4.1.c Unpacking Packed Values

• Unpacking is the splitting packed values into individual variables - a, b, c = [1, 2, 3], here a, b, c is a tuple of 3 variables
• In a function, positional arguments assigned to parameters gets unpacked into a tuple.
• Unpacking works on any ==iterable== type.

4.1.d Simple Application of Unpacking (Swapping values in two vars)

• In Part 2, The right hand side is evaluated completely first.
  1. The RHS constructs a tuple, where
  • first element references 0x1111
  • second element references 0x1000
  2. A new tuple object (y, x) is created and stored in a new memory address
  3. And then the reference to Object(new tuple at 0x1345) is assigned to tuple(x, y)
• So when swapping variables in python, there is no change in value.
  1. new tuple with swapped references are created
  2. then the original tuple is set to reference the newly created tuple

# Traditional Approach
x, y = 10, 20
temp = x
x = y
y = temp

# Python Approach
'''
This works because, in python:
1. The entire RHS is evaluated first and completely
2. Then the assignments are made

Example: Let's say we have to swap these values

a,b = 1,2




'''
x, y = 10, 20
x, y = y, x

10 20
20 10

4.1.e Unpacking Sets & Dictionaries

d = { "key1": 1, "key2": 2, "key3": 3}

# Iterates over the keys in the dictionary by default
print([i for i in d])

# Unpacks the keys of the dict
a, b, c = d
print(a, b, c)

s = {1,2,3,4,5}
print([i for i in s])

['key1', 'key2', 'key3']
key1 key2 key3
[1, 2, 3, 4, 5]
0 1
1 2
2 3
3 4
4 5

4.2 Extended Unpacking

4.2.a Uses of * operator - the rest of the elements operator

4.2.a.i * operator for ordered lists

# In the LHS
a, *b  = [1, 2, 3, 4, 5]        # a=1, b=[2,3,4,5]
a, *b  = (1, 2, 3, 4, 5)        # a=1, b=(2,3,4,5)
a, *b = "ABCDE"                 # a="A", b=["B","C","D","E"]
a, *b, c = "ABCDE"              # a="A", b=["B","C","D"], c="E"
a, b, *c, d = "ABCDE"           # a="A", b="B", c=["C","D"], d=["E"]

# In the RHS
l1 = [1,2,3]
l2 = [4,5,6]
l3 = "XYZ"

l1_l2 = [*l1, *l2]              # l1_l2 = [1,2,3, 4,5,6]
l2_l3 = [*l2, *l3]              # l2_l3 = [4,5,6, "X","Y","Z"]

• * operator denotes the rest of the elements.
• It can only be used once in the LHS of an unpacking assignment.

4.2.a.i * operator for unordered lists

# In the LHS
s = {1,2,3,4,5}
a,b,*c = s                      # a=1, b=2, c=[3,4,5]

d = {
    "key1": 1,
    "key2": 2,
    "key3": 3
}
a,*b = d                        # a="key1", b=["key2", "key3"]
# But since sets and dicts are supposed to be unordered, this is generally not recommended


# In the RHS
d1 = {
    "key_a": 1,
    "key_b": 2,
    "key1": 10,
}
keys = [*d, *d1]             # keys = ["key1","key2","key3","key_a","key_b","key1"]
keys = {*d, *d1}             # Same as above but duplicate keys are only included once

4.2.b ** unpacking operator - unpack key-value pairs

• ** cannot be used in the LHS of an assignment

d1 = {'p':1, 'y':2}
d2 = {'t':3, 'h':3}
d3 = {'h':5, 'o':6, 'n':7}

merged = {**d1, **d2, **d3}
'''
Since, the key 'h' is repeating, the values of the 'h' key gets overridden to the latest value - rightmost value
'''
print(merged)

d1 = {'a':1, 'b':2}
new_d = {'a':3, 'b':4, **d1}    #-> {'a':1, 'b':2}


new_d1 = {**d1, 'a':3, 'b':4}   #-> {'a':3, 'b':4}

{'p': 1, 'y': 2, 't': 3, 'h': 5, 'o': 6, 'n': 7}

4.2.c Nested unpacking

l = [1,2, [3,4]]
a,b,(c,d) = l
print(a,b,c,d)

l = [1,2, "XYZ"]
a,b,(c,d,e) = l
print(a,b,c,d,e)

# Using multiple * operators inside nested unpacking stmts

l = [1,2,3,"python"]
a, *b, (c, *d) = l
print(a,b,c,d)

# Using indices & slices
a,b,c,d = l[0], l[1:-1], l[-1][0], list(l[-1][1:])
print(a,b,c,d)

1 2 3 4
1 2 X Y Z
1 [2, 3] p ['y', 't', 'h', 'o', 'n']
1 [2, 3] p ['y', 't', 'h', 'o', 'n']

# a,b = 1,2  and a,b, = 1,2 are syntatically the same

a, b = 1,2      # a=1,b=2
print(a, b)

a,*b = 1,2,3    # a=1,b=[2,3]
print(a, b)

a,*b = 1,       # a=1,b=[]
print(a, b)

*a, b = 1,      # a=[],b=1
print(a,b)

b, *a = 1,      # a=[],a=1
print(a,b)

*a, b = 1,2     # a=[1],b=2
print(a,b)

1 2
1 [2, 3]
1 []
[] 1
[] 1
[1] 2

4.3 agrs in Python

• *args can be used to pass in variable number of positional arguments into functions.
• *args exhausts ==positional argument== - other args cannot be written after *args
  • def func(a,b,*c,d): - here c will unpack all the positional arguments from the 3rd position and d will not get anything.
• *args returns a tuple

def my_func(a,b,*c): # c ends up a tuple, not a list - unlike regular unpacking
	print(a,b,c)

my_func(1,2,3,4,5)
my_func(1,2, "python", "value")

def std_practise(a,b,*args):
	pass

def unpacking_rhs(a,b,c):
	print(a,b,c)

l = [1,2,3]
unpacking_rhs(*l)

1 2 (3, 4, 5)
1 2 ('python', 'value')

4.4 keyword arguments in Python

• when using *args, it is important to remember that it only catches positional arguments

def my_func(*args, d):
	print(args, d)

my_func(1,2,3, d=10)

try:
	my_func(1,2,3,10)
except TypeError:
	print("all the positional args are captured by *args, nothing remaining for 'd'")

(1, 2, 3) 10
all the positional args are captured by *args, nothing remaining for 'd'

4.4.a omitting any mandatory positional argument using *

# here the function doesnot allow any positional arguements to be passed
# only the keyword arg called d is allowed
def func(*, d):
	print(d)

try:
	func(1,2,3,d=10)
except TypeError:
	print("No positional arguements allowed")

func(d=10) # Allowed

No positional arguements allowed
10

4.4.b Putting it all together

def function(a, b=1, *args, d, e=True):

• a - mandatory positional argument (may be specified using named argument)
• b - optional positional argument (because of the default value of 1)
• args - catch-all for any additional positional argument(optional)
• d - mandatory keyword argument (because *args catches all the other positional argument)
• e - optional keyword argument (because of the default value of True)

def function(a, b=1, *, d, e=True):

• a - mandatory positional argument (may be specified using named argument)
• b - optional positional argument (because of the default value of 1)
• * - any other positional arguments are discarded
• d - mandatory keyword argument (because *args catches all the other positional argument)
• e - optional keyword argument (because of the default value of True)

4.5 kwargs(keyword arguments) in Python

• **kwargs catches all the remaining keyword arguments provided
• ** is the main spotlight here
• No parameters can come after the **kwargs - it is the end of the line

def my_func(d,e, **kwargs):
    print(d,e,kwargs)

my_func(d=10,e=20, a='10', b='20')

def func(*,d,**kwargs):
    print(d,kwargs)

func(d=10, a='10', b='20')

10 20 {'a': '10', 'b': '20'}
10 {'a': '10', 'b': '20'}

4.6 Writing a generic timer

import time

def time_it(fn, *args, rep=1, **kwargs):
    '''
    Why is args & kwargs not passed directly ?

    Example:

    def time_it(fn, *args, **kwargs):
        fn(args, kwargs)

    def function():
        pass

    time_it(function, 1,2, a=10, b=20) will execute and the passed params will look like:

    function( (1,2), {"a"=10, "b"=20} )

    But we want this:
    function(1,2, a=10, b=20)

    Changes we need are:
    - unpack the tuple into its elemtns - *args
    - unpack the dict into its key-val pair items - **kwargs
    '''
    start = time.perf_counter()
    for i in range(rep):
        fn(*args, **kwargs)
    end = time.perf_counter()
    return (end - start) / rep # return the average execution time

4.7 Gotchas for using default arguments

4.7.a What happens in runtime in python

from datetime import datetime

def log(msg, * , timestamp=datetime.now()):
    print(msg, "at", timestamp)

log("log #1")
print("sleeping for 1 sec")
log("log #2")

log #1 at 2025-09-27 11:56:20.235224
sleeping for 1 sec
log #2 at 2025-09-27 11:56:20.235224

• How is utcnow() timestamp same in two separate function calls ?
• What is happening at runtime ?
  • At the start-up the def gets executed once - creating the function Object in the memory.
  • During that Object creation timestamp=datetime.utcnow() is evaluated - this doesn’t happen during function calls, it is a one and done execution.

# Solution
from datetime import datetime

def log(msg, * , timestamp=None):
    if not timestamp:
        timestamp = datetime.now()
    print(msg, "at", timestamp)

log("log #1")
print("sleeping for 1 sec")
log("log #2")

log #1 at 2025-09-27 11:56:32.542532
sleeping for 1 sec
log #2 at 2025-09-27 11:56:32.542536

4.7.b Mutable Default Arguments

• Default Arguments are evaluated once, during the execution of the def statement, and never after during the programs the execution.

def add_to_list(item , list=[]):
    list.append(item)
    return list

print(add_to_list(1))
print(add_to_list(1))

# Solution
def add_to_list(item , list=None):
    if not list:
        list = []
    list.append(item)
    return list

print(add_to_list(1))
print(add_to_list(1))

[1]
[1, 1]
[1]
[1]

4.7.c Using Default Argument Gotchas to your advantage

• Default Argument Gotchas can be used as a memoization mechanism in recursive functions

def factorial(n, *, cache={}):
    if n<1:
        return 1
    elif n in cache:
        print(f"found {n}!")
        return cache[n]
    else:
        print(f"calculating {n}!")
        result = n * factorial(n-1)
        cache[n] = result
        return result

print(factorial(2))
print(factorial(3))

calculating 2!
calculating 1!
2
calculating 3!
found 2!
6

5. First Class functions & Higher Order Functions

• What are first class objects?

  • Can be passed to a function as an argument.
  • Can be returned from a function
  • Can be assigned to a variable
  • Can be stored in a data-structure - lists, tuples, sets, dicts etc

• What are higher Order Functions?

  • takes a function as an arg
  • returns a function

5.1 Docstrings & Annotations

5.1.a Docstrings

• Any kind of string lit in the first line of a function is called a Docstring.
• __doc__ property of the function object contains the Docstring
• Defined in PEP 257 of the documentation.

def fact(n):
	'''Calculates the fact of the n!'''
	pass

print(fact.__doc__)

Calculates the fact of the n!

5.1.b Annotations

• Defined in PEP 3107
• Additional ways to document functions.
• This is just metadata - doesn’t affect how python code runs
• Annotation can be any expression - which means it can be strings, types or anything

def my_func(a: 'a string', b: 'a positive integer') -> 'a string':
    return a * b

help(my_func)

def my_func(a: str, b: int) -> str:
    return a * b

x = 0
y = 10
def my_func(a:str) -> 'a repeated ' + str(max(x,y)) + ' times': # all expressions in annotations run once during the def stmt execution
    return a * max(x,y)

print(my_func.__annotations__)
help(my_func)

Help on function my_func in module __main__:

my_func(a: 'a string', b: 'a positive integer') -> 'a string'

{'a': <class 'str'>, 'return': 'a repeated 10 times'}
Help on function my_func in module __main__:

my_func(a: str) -> 'a repeated 10 times'

5.2 Lambda Expressions

• Lambda Expressions are an alternative way of defining functions - without a name

• lambda [param list]: expression - this statement returns a function object

• lambdas, anonymous functions are not equivalent to closures

• Limitations

  • The “body” of a lambda is limited to a ==single expression== - has to be a single logical line of code
  • No assignments in the body
  • No annotations

func = lambda x : x

def func(x):
    return x

# lam_func and def_fun are exactly the same

5.2.1 Lambda & Sorting

• sorted can be used to sort iterables

import random

l = [2,5,2,6,8,9,8,4,22,1,5,8]

print(sorted(l))

# reverse sorting
print(sorted(l, key=lambda e: -e))

# Sorting the keys using the value of dict
d = {"key1":9, "key2":2, "key3":8}
print(sorted(d, key=lambda e: d[e]))

ll = [ "asd", "xzc", "asd", "re", "por"]
print(sorted(ll, key=lambda s: s[-1]))

print(sorted(l, key=lambda e: random.randint(0,100)))

[1, 2, 2, 4, 5, 5, 6, 8, 8, 8, 9, 22]
[22, 9, 8, 8, 8, 6, 5, 5, 4, 2, 2, 1]
['key2', 'key3', 'key1']
['xzc', 'asd', 'asd', 're', 'por']
[22, 8, 4, 8, 2, 2, 9, 5, 6, 1, 5, 8]

5.3 Callables

• Any object that can be called using a () operator.
• Always returns a value - returned value can be None.
• Check an object is callable using the callable(obj) function

5.4 Higher Order Functions - Map, Zip, Filter

• Functions that takes function as arguments or return function are called higer-order functions

5.4.1 The map function

• map(func, *iterables) returns an iterator that calculates the function applied to each element.
  • func - the calculation function that takes number of arguments equal to number of iterables provided.
  • *iterables - variable number of iterable objects.
• The iterator stops as soon as one of the iterables end.

l = [2,3,4]
m = [1,2,3,4]

print(list(
    map(lambda x: x**2 , l)
))

print(list(
    map(lambda x,y: x+y, l,m)
))

[4, 9, 16]
[3, 5, 7]

5.4.2 The filter function

• filter(func, iterable) - returns an iterator with the elements of the iterable for which the func returns a Truthy Object.
  • func - a function which takes a single arg.
  • iterable - a single iterable

l = [0,1,2,3,4,5]

# This works because, filter works with truthy objects, and 0 is non-Truthy and rest of the items are truthy
print(list(
    filter(None, l)
))

[1, 2, 3, 4, 5]

5.4.3 The zip function

• zip(*iterables) returns an iterator that combines multiple iterables.
  • *iterables - variable number of iterable objects.
• The zip stops as soon as one of the iterables end.

l = [1,2,3]
l1 = [10, 20, 30 ,40]
l2 = 'python'

print(list(
    zip(l,l1,l2)
))

[(1, 10, 'p'), (2, 20, 'y'), (3, 30, 't')]

5.4.4 List Comprehension - an alternative to maps

# getting squares of all the elemtns

l = [2,3,4,5]
m = [1,2,3,4]

print([i**2 for i in l])

print([i+j for i,j in zip(l,m)])

[4, 9, 16, 25]
[3, 5, 7, 9]

5.4.5 List Comprehension - an alternative to filter

# getting squares of all the elemtns

l = [2,3,4,5]

print([x for x in l if x%2 == 0 ])

[2, 4]

5.4.6 map, filter, zip gotchas

def factorial(n):
    if n<1:
        return 1
    else:
        return n * factorial(n-1)

l = [1,2,3,4,5]

facts = map(factorial, l)

for i in facts:
    print(i)

# This print doesn't give any values.
# Because map, filter, zip return generator objects - which only give us values when we start the iteration.
# Values are not calculated and stored at the time of calling map()
# But rather generator objects are created to generate those values
for i in facts:
    print(i)

for i in (factorial(x) for x in l):
    print(i)

1
2
6
24
120
1
2
6
24
120

5.4.7 Reducing Functions

• functions that recombine iterable recursively, ending up with a single return value.
• also called accumulators, aggregators or folding functions.

# A reducer function example

l = [1,2,3,4,5,6,7,8,9]

def _reduce(fn ,seq):
	result = seq[0]
	for x in seq[1:]:
		result = fn(result,x)
	return result

add = lambda a,b:a+b
_max = lambda a,b: a if a>b else b

print(_reduce(add, l))
print(_reduce(_max, l))

45
9

• reducing using the std module
  • min - returns smalles
  • max - returns largest
  • sum - sum of all elements
  • any - returns True if any element is truthy
  • all - returns True if all elements are truthy

from functools import reduce

l = [5,8,6,10,9]

print(reduce(lambda x,y: x*y, range(1,6)))
print(reduce(lambda x,y: y if y>x else x, l))
print(max(l))

120
10
10

6. Scopes, Closures & Decorators

6.1 Scopes and Namespaces

6.1.a Global Scope

• A module scope is called the global scope

• Spans single file

• The only truly global scope is the ==built-in== scope

• lexical scope is stored in a table called namespace

• Example:

  • In module1.py - If we say print(True)
    • python looks for print function and True object in the local scope - i.e in module1.py
    • If it doesn’t find it python searches in the outer-scope and keeps searching like that and finally arrives at the built-in scope - where True and print is defined

6.1.b Accessing global scope from local scope

• Global scope variable is only shadowed when an assignment is used in the local scope.
• Similar to the global keyword, if an assignment operation is present anywhere in the local scope for the variable - it is considered local by python.

a = 0

def shadowed():
	a=10
	pass

def not_shadowed():
	b = 1 + a
	pass

# But there is a caviat

def my_funct():
	print("global a: ", a)
	a = 200
	print(a)

'''
	You would expecte this to behave like this:
	Output>>

	global a: 0
	200

	Similar to the "global" keyword, if an assignment is present anywhere in the local scope.
	The variable will be considered local by python.
'''
try:
	my_funct()
except UnboundLocalError:
	print("local var 'a' referenced before assignment")

local var 'a' referenced before assignment

• when you use the global keyword, the variable doesn’t have to exist previously in the global scope
• using global var keyword in the local scope will always reference the local variable as global - no matter the line at which global var is written.

a = 0

def my_func():
    a = 100 # this a is not using the a from the global scope
    print(a)

my_func()
print(a)


def my_func():
    global a
    a = 200 # After explicit specification python treats this a as the same as global namespace a
    print(a)

my_func()
print(a)

  File "/var/folders/fk/rxb7n76s4gng0s0j1s2ll98c0000gn/T/mdlab/mdlab.py", line 806
    print(''', flush=True)
                         ^
SyntaxError: unmatched ')'

6.2 Nonlocal scope

• nonlocal keyword can be used to access nonlocal scope variable.
• nonlocal select the first matching name - when going out from inner most scope
• assignment expression, without the presence of nonlocal var stmt - means the variable var = 10 is a local variable

a = 10

def outer():
    a = 20
    def inner():
        a = 30 # python sees assignment and makes "a" a local scope var


def outer():
    a = 20
    def inner():
        nonlocal a # python sees "nonlocal" and makes "a" a non_local scope var - i.e outer scope var
        a = 30

def outer():
    a = 20
    def inner():
        global a # python sees "global" and makes "a" a global scope var
        a = 30

  File "/var/folders/fk/rxb7n76s4gng0s0j1s2ll98c0000gn/T/mdlab/mdlab.py", line 806
    print(''', flush=True)
                         ^
SyntaxError: unmatched ')'

6.3 Closures

• Closures are used when there are shared variables between multiple scopes
• Closure = inner + extended scope x

def outer():
    x = "hello"
    def inner():
        print("the value is: ", x) # here x is
    return inner # when inner returns, actually a cloure is being returned

# Ater calling outer, x goes out of scope and should be removed
fn = outer()
# How does inner function call still have x, availabe to print
fn()

the value is:  hello

6.3.1 Python Cells and Multi-scoped Variable

• Whenever there is a multi-scoped variable - nonlocal or closure.
• Python uses an Object called “Cell”

• After the completion of outer(),outer.x` is cleared.
• But the inner.x still has a reference to the Object(cell) - thus it is not cleared.
• Multi-scope variable do a double-hop to get the value - which only happens when inner is called
  • One to the cell
  • From cell to the value

def outer():
    x = "hello"
    print("outer", hex(id(x))) # python automatically takes care of the double-hop & returns the indirect refrence

    def inner():
        print("inner", hex(id(x))) # python automatically takes care of the double-hop & returns the indirect refrence
        print("the value is: ", x)
    return inner

fn1 = outer()
fn1()

print(fn1.__closure__)
print(fn1.__code__.co_freevars)


fn2 = outer()
fn2()

print(fn2.__closure__)
print(fn2.__code__.co_freevars)

# fn1 & fn2 have different instances of a closure

outer 0x100fc91a0
inner 0x100fc91a0
the value is:  hello
(<cell at 0x101e97970: str object at 0x100fc91a0>,)
('x',)
outer 0x100fc91a0
inner 0x100fc91a0
the value is:  hello
(<cell at 0x101e979d0: str object at 0x100fc91a0>,)
('x',)

6.3.2 Shared Extended Scopes & Late Binding gotchas

import functools

adders = []
for n in range(1,4):
    adders.append(lambda x: x+n)

'''
n = 1: a closure with free variable 'n' -> pointing to cell -> pointing to 1 is created
n = 2: a closure with free variable 'n' -> pointing to the same cell -> now pointing to 2 is created
n = 3: a closure with free variable 'n' -> pointing to the same cell -> now pointing to 3 is created

- At the end of the loop, all the free variable 'n's are pointing to the same cell.
- That cell is pointing to 3
- i.e all free variables 'n' are pointing to 3 indirectly

'''
print(adders[0](10)) # 13
print(adders[1](10)) # 13
print(adders[2](10)) # 13

# Solutions
adders = []
for n in range(1,4):
    adders.append(lambda x, c=n: x+c)

adders = []
for n in range(1,4):
    adder = functools.partial(lambda x, c: x+c, n)
    adders.append(adder)

13
13
13
11
12
13
11
12
13