Python is a programming language which was first released in 1991. It has stood the test of time and after nearly 30 years it is still widely used in software industry. The simplicity and concise syntax of python with growing number of libraries and framework still makes it one of the language of choice. Today python is used in web scraping, data science, machine learning, image processing, NLP, data processing and many more areas.
Python language is a specification and has multiple implementations of it e.g. CPython (widely used standard implementation), PyPy, IronPython, Jython. IronPython is python implementation written in C# targeting Microsoft’s .NET framework and uses .Net Virtual Machine for execution. PyPy is python implementation which is faster than Python as it uses Just-in-Time compiler to translate Python code directly into machine-native assembly language. Jython is an implementation of the Python programming language that can run on the Java platform. Jython programs use Java classes instead of Python modules. Jython compiles into Java byte code, which can then be run by Java virtual machine. Jython enables the use of Java class library functions from the Python program.
Python code is first compiled by compiler to byte code which is later interpreted by Python virtual machine to the machine language. The .py source code is first compiled to byte code as .pyc. The byte-code are the instructions similar in spirit to CPU instructions, but instead of being executed by the CPU, they are executed by the Python Virtual Machine (PVM).
Statement, Indentation and Comments
In Python, the end of a statement is marked by a newline character. But we can make a statement extend over multiple lines with the line continuation character (\) as below.
a = 1 + 2 + 3 + \
4 + 5 + 6 + \
7 + 8 + 9
Line continuation is implied inside parentheses ( ), brackets [ ], and braces { }. We can also put multiple statements in a single line using semicolons, as follows:
a = 1; b = 2; c = 3
Most of the programming languages like C, C++, and Java use braces { } to define a block of code. Python, however, uses indentation. Indentation refers to the spaces at the beginning of a code line. A code block (body of a function, loop, etc.) starts with indentation and ends with the first unindented line. The amount of indentation is up to the developer, but it must be consistent throughout the block. Generally, four whitespaces are used for indentation and are preferred over tabs. PEP 8 is the documentation which has all the best practices for formatting the code.
In Python, the hash (#) symbol is used to start writing a comment which extends until the newline character. For multi-line comments which extend up to multiple lines, the triple quotes either ''' or """ is used as below.
# this is a python comment """ This is Documentation comment for python """
Data Types
Python supports Dynamic Typing of the variable were type depends on the value assigned to the variable. When a variable is assigned a value, actually it's a label pointing to the memory location with the value. If the variable is assigned to a new value, the type of the variable changes depending on the type of the value it's pointing to. Python is case sensitive language, hence uppercase variable is different from lowercase variable. Generally lower case is used for variable names, and "_" (underscore) is used to separate multiple words in variable or function name instead of camel case.
x = 5 # type of variable x is integer x = 'Example' # type of variable x is now string
In Python every value is an object. A variable is a label pointing to the particular object (with memory location). When a variable is assigned a value 100, the label of the variable is pointing to the object containing value 100. If another variable is assigned same value 100, it will also point to the same object with 100. As more and more variable have the same value, the reference count to the object 100 increases. The reference count to the object decreases when the variables are re-assigned a different value, the variable goes out of scope or when del keyword is used to remove the variable (label) as a reference to the object. Once all the references to the object are removed, it can then be safely removed from the memory. Internally the python object holds the object type, its reference count and its value. If no variables is pointing to any given value or address location, then python makes it ready for garbage collection. Python uses both reference counting as well as generational garbage collection (variation of mark and sweep).
There is no concept of constant value in python. It depends on programmer to not change the value and treat it as a constant.
None: It represents no value assigned to variable. It is similar to null defined in other languages.
Numeric: It has 4 types namely: Int is integer, Float is floating point, Bool (True/False) and Complex. The boolean value of True (int(True) = 1) is 1 and boolean value for False is 0. Complex numbers have a real and imaginary part e.g. complex_number = 6 + 9j. Complex number can also be created using complex(b, k), were b is integer and k is floating point number.
Python has functions to create integer, boolean, float variables from string as below.
int_val = int("45")
float_val = float("6.78")
bool_val = bool("true")
Sequence types include List, Tuple, Set, String and Range which are discussed below. Apart from these python also has a Dictionary similar to a HashMap in Java.
Python provides the built-in type() function, which when passed one argument, returns the type of an object. It is used to get the type of the variable i.e. Integer, Float, String or Boolean.
type(var_name)
Standard Functions
Python has a set of built in functions.
The print() function is a standard function to print variables. It also takes expressions to print the results.
print('*' * 10) # using expression within print function
By default the print function prints the text in a new line. To print the text in the same current line, we pass second argument end="" as below:
print('*' * 10, end="")
The input() function is used to get the user input in python.
field = input("Enter the value: ")
The eval() function is used to evaluate an expression or execute a function and return the corresponding result.
eval(input("Enter an expression: "))
# When entered the expression (2 + 6 - 1), it prints the result 7
Argument values "argv" is used to get arguments passed while running the python program.
import sys x = int(sys.argv[1]) y = int(sys.argv[2]) z = x + y print(z)
The id() function is used to get the address of the value pointed by the variable.
var_name = "Example" id(var_name)In python if multiple variables have the same data then all the variables will point to the same single address which contains the same data. In the below example the address pointed by a, b and 100 is the same, i.e. id(a) == id(b) == id(100). Further as we change the value of a to 200, the value of b remains the same i.e. 100, even if we stated (b = a) above.
a = 100 b = a cond1 = (id(a) == id(b)) # returns true print(a is b) # prints true, as both variable a and b point to same value or object cond2 = (id(b) == id(100)) # returns true a = 200 cond3 = (b == 100) # returns true, even though we declared b=a and changed a to 200
String
Strings are amongst the most popular types in Python. We can create them simply by enclosing characters in quotes. Python treats single quotes the same as double quotes. Creating strings is as simple as assigning a value to a variable. Python support below special operators for string.
| Operator | Description | Example |
|---|---|---|
| + | Concatenation - Adds values on either side of the operator | a + b will give HelloPython |
| * | Repetition - Creates new strings, concatenating multiple copies of the same string | a*2 will give -HelloHello |
| [] | Slice - Gives the character from the given index | a[1] will give e |
| [ : ] | Range Slice - Gives the characters from the given range | a[1:4] will give ell |
| in | Membership - Returns true if a character exists in the given string | H in a will give 1 |
| not in | Membership - Returns true if a character does not exist in the given string | M not in a will give 1 |
| r/R | Raw String - Suppresses actual meaning of Escape characters. The syntax for raw strings is exactly the same as for normal strings with the exception of the raw string operator, the letter "r," which precedes the quotation marks. The "r" can be lowercase (r) or uppercase (R) and must be placed immediately preceding the first quote mark. | print r'\n' prints \n and print R'\n'prints \n |
| % | Format - Performs String formatting | print "My name is %s and weight is %d kg!" % ('Zara', 21) |
str = "Python"
new_str = "I Like" + str
print( str[-1] ) # 1st character from the end
print( str[0:3] ) # Returns all chars from 0 till 3, not including the 3rd char
print( str[2:] ) # Returns all chars from 2 till (default) end of the string
print( str[:5] ) # Returns all chars from 0 (default) till 5 (excluding 5th char)
another = str[:] # Returns all the characters from the string to create a clone of string
another = str[1:-1] # Returns all chars from index 1 till 1st char from the end (excluding the last char)
# Trying to modify char in string throws error as strings are immutable.
str[0] = 'R'
# example of formatted string with prefix 'f' to use formatted strings
sentiment = 'good'
msg = f'{str} is a {sentiment} language'
# example of a identated string paragraph
para = '''
used for paragraph. The text is printed with indentation as it is in the source code.
'''
print(para)
The len() and print() are general purpose functions in python which work with other types as well. The rest of the below functions such as find(), replace(), upper() etc are string specific functions.
To check if the string contains a given word or not, we use the 'in' operator which returns a boolean value, below is the example expression.
len(str) # returns the no of characters in the string.
str.isupper() # checks if string is in upper case
# Here upper() is a method, while len(), input() are general purpose functions. It creates a new string without modifying the existing string.
upper_str = str.upper()
# Returns index of the first occurrence of the character or string. It is case sensitive.
str.find('y')
paragraph = "I like programming in Python"
str.find('Python') # returns index of the word python in the string paragraph.
str.replace('Python', 'Java') # replaces the word with the passed word in the string
str.replace('P', 'J') # replace a given character with another character
string.split(' ') # returns a list of items (words)
text = "Its fun to program in Python." 'Python' in text
The title() method returns a string where the first character in every word is upper case. Like a header, or a title. If the word contains a number or a symbol, the first letter after that will be converted to upper case.
Binary and Decimal Numbers
The bin() function converts decimal number to binary format. The '0b' prefix symbolizes that the number is in binary format.
bin(25) = 0b11001
For octal format we use oct() function which returns result with prefix '0o'. Similarly for hexadecimal we use hex() function which returns the results with prefix '0x'.
Below are the bitwise operators in Python:
# compliment of number ~12 = -13 # bitwise and operator 12 & 13 = 12 # bitwise or operator 12 | 13 = 13 # bitwise ex-or operator 12 ^ 13 = 1 # bitwise left shift operator, which shift 2 bits to left 10 << 2 = 14
Arithmetic Operators
Python supports all the standard arithmetic operations (+,-, *). Although it has two types of division operators as below.
# Normal division which returns floating point number 10 / 3 = 3.33333333 # Whole division which return an integer result. 10 // 3 = 3
Exponent operator
10 ** 3 = 1000 # means 10 to the power 3
Augmented assignment operators
x += 3 x -= 3
Operator Precedence: Below is the order of precedence for operators in Python
- Parentithesis
- Exponentiation (12 ** 3)
- Multiplication or Division
- Addition or Subtraction
Python also allows assignment of multiple variables to different values.
a, b = 1, 2
It also allows to swap two variables as below: Python uses ROT_TWO() to swap the the two top most stack items for this operation.
a, b = b, a
Math Functions
Python support basic math operations by default as below.
round(2.9) = 2 # rounds a float value to integer abs(-2.9) = 2.9 # returns positive representation of the value
It supports mathematical functions after importing the Math module.
import math math.ceil(2.9) = 3 # Get ceiling of the number math.floor(2.9) = 2 # Get floor of the number math.sqrt(25) # returns square root of the number math.pow(3,2) = 9 # power function math.pi # value of PI # import module and define module alias import math as m m.sqrt(25)
Random integer values can be generated using the random() function which also allows to specify the range for the random values.
import random random.random() random.randint(10, 20)It also allows to select a random item from a list.
members = ["John", "Sam", "Michael", "Arthur", "George"] random_member = random.choice(members)
If Else Statement
The if and else statements are used for conditional execution.
if a == true:
print("Boolean value is true")
print("Second line")
elif b == true:
print("Another Boolean value is true")
else:
print("Boolean value is false")
# if condition can also we written as a single line conditional assignment statement
fruit = "Apple"
is_apple = True if fruit == 'Apple' else False
Python support all basic comparison operators including <, >, <=, >=, ==, !=. It also provides following logical operators to combine multiple conditions.
- and: if a and b
- or: if a or b
- not: if a and not b
While Loop
The while statement is used for repeated execution as long as an expression is true.
i = 1 while i < 6: print(i) i += 1While statements in Python can optionally also have an else block, which is executed only when the while loop completes successfully without any breaks. As the while loop can be terminated with a break statement, in which case the else part is ignored. Hence, a while loop's else part runs if no break occurs and the condition is false.
while counter < 3:
print("Inside loop")
counter = counter + 1
else:
print("Inside else")
For Loop
For loop allows to iterate over the elements of a sequence (such as a string, tuple or list) or other iterable object.
for item in "Sample String": print(item) for item in ["Toyota", "Honda", "Ferrari"]: print(item)The for loop containing a condition in python has an else block which essentially means if nothing matched and no break occurred then execute the else block outside the for loop. The break condition is important here as without break, it will execute the else block every time.
for num in [12, 34, 23, 67]
if num % 5 == 0:
print(num)
break
else:
print("Not Found")
The range function creates objects based on the from/to parameters passed to it. Range object can also be converted to a list by passing it to list() function e.g. list(range(10)). Below is the example of range functions.
range(10) # returns numbers from 0 to 9 range(5, 10) # returns numbers from 5 to 9 range(5, 10, 2) # returns numbers from 5 to 9 with steps of +2 range(20, 10, -1) # returns numbers from 20 to 10 in descending order # for loop can be used to loop through the values within the range for item in range(10) print(item)
An underscore '_' can also be used as a variable in both for and while loops.
_ = 5
while _ < 10:
print(_, end = ' ') # default value of 'end' id '\n' in python. we're changing it to space
_ += 1
for _ in range(5):
print(_)
Python support keywords, break, continue and pass. Break statement breaks the loop and continue skips the remaining code continuing with the loop. Pass keyword indicates that the block (can be a if/else block, loop block, class/method block) is empty and to skip it.
Arrays
Arrays is similar to a list were all the items are of same type. Arrays in python don't have specific fixed size. Arrays can be created using array(type, list of values) method. The type value is taken from python type code.
from array import *
vals = array('i', [5,9,8,4,2]) # type value 'i' indicates signed int
print(vals)
vals.buffer_info() # returns a tuple (address, size), with address and size of the array.
vals.typecode # returns i (signed integer)
vals.reverse() # reverse the values of the array
vals.index(2) # returns index of element 2 i.e. 4
# append() method is used to add values to the array.
vals.append(10)
empty_array = array('i', []) # empty array of integers
newarray = array(vals.typecode, (a for a in vals))
for e in vals:
print(e)
Lists
List can contain items of different types as opposed to an array.
names = ["John", "Bill", "Tony"]
names[-1] # returns Tony
names[1:] # returns a new list from 1st element (0th being the first) to the end of the list
names[1] = "Billy" # update 2nd element
matrix = [ [1,2,3], [5,6,7], [3,7,8] ]
for row in matrix:
for item in row:
print(item)
numbers = [4,6,8,10]
numbers.append(56) # adds item to the list
numbers.insert(index, item)
numbers.extends([16,37,68]) # add multiple elements to list
numbers.remove(item)
numbers.clear()
numbers.pop() # remove last item
numbers.pop(index) # remove indexed item
numbers.index(item) # returns index of the first occurrence of the item
numbers.count(item) # number of times the item occurs within the list
min(numbers) # get minimum value in the list
sum(numbers) # get sum of all items in the list
50 in numbers # existence of an item within the list
50 not in numbers # 50 does not present in the list then true
numbers.sort()
numbers.reverse() # reverses the list
numbers_copy = numbers.copy() # create a copy of the list
List Comprehensions allows conditional construction of list literals using for and if clauses. They provide a more concise way to create lists.
print [i for i in range(10)] # [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] print [i for i in range(20) if i%2 == 0] # [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]
Enumerate Function
The enumerate() function assigns an index to each item in an iterable object that can be used to reference the item later. The enumerate() function takes an iterable type object like a list, tuple, or a string, and an optional start parameter (0 by default) which tells which index to use for the first item of the iterable object. By default, the enumerate() function returns an enumerated object which can be converted to a tuple or a list by using tuple(<enumerate>) and list(<enumerate>), respectively.
cities = ['Amsterdam', 'New York', 'Paris', 'London']
for i, city in enumerate(cities)
print(i, city)
The enumerate() function can be used instead of the for loop. That’s because enumerate() can iterate over the index of an item, as well as the item itself.
cars = ['kia', 'audi', 'bmw'] print(list(enumerate(cars, start = 1)))
Tuples
Tuples are same a list but unlike lists are immutable.
numbers = (1, 2, 3)Tuples does not allow to change it items once assigned, as it does not support object assignment (hence they are immutable). It has only two methods (similar to lists) index() and count(). To read a specific item in the tuple use :
print(numbers[0])When returning tuples from a function, we don't need to specify the brackets, e.g.
return a, b # Python automatically interprets this as a tupleUnpacking feature: Used to assign items of lists or tuples to variables by unpacking them.
coordinates = (1, 2, 3) x = coordinates[0] y = coordinates[1] z = coordinates[2] x, y, z = coordinates list = (1, 2, 3) a, b, c = listIf we don't want to use specific values while unpacking, just assign that value to special underscore '_' variable.
# ignoring a value a, _, b = (1, 2, 3) # a = 1, b = 3 # ignoring multiple values # *(variable) used to assign multiple value to a variable as list while unpacking # it's called "Extended Unpacking", only available in Python 3.x a, *_, b = (7, 6, 5, 4, 3, 2, 1)
Sets
Set does not retain the order of elements as it uses hash. Hence indexes are not supported in set. It does supports add(), remove(), pop() methods to update the set. Also there are no duplicates allowed in a set.
s = { 3, 4, 5, 7, 8, 18 }
Dictionary
Dictionary is of mapping data type. Dictionaries are used to store information as key value pairs. Every key is unique and should be immutable within a given dictionary.
customer = {
"name": "John",
"age": 67,
"is_retired": False,,
"cars": ["Honda", "Toyota"]
}
# returns value "John" associated with the "name" key in customer dictionary
# If specified key does not exits then it throws an error.
customer["name"]
# get() method also fetches the value of the key from the dictionary
# if the key is not present in dictionary then it returns None without throwing an error
customer.get("name")
# Also allows to specify default value which will be returned when no key exists within the dictionary
customer.get("name", "default_value")
customer["name"] = "Jack" # updates the value of the key "name" in the dictionary
customer["birth_date"] = "10 Jan 1980" # Adds new key-value to the dictionary
customer.clear() # empty the dictionary
customer.keys() # returns all keys in the dictionary
customer.values() # returns all values in the dictionary
The del keyword is used to delete objects. In Python everything is an object, so the del keyword can also be used to delete variables, lists, or parts of a list etc.
del customer["birth_date"]
Zip Function
The zip() function creates an iterator of tuples which aggregate elements from two or more iterables. It pairs the first item in each passed iterator together, followed by paring the the second item in each passed iterator together and so forth.
keys = ["Texas", "Ohio", "California"]
values = ["Austin", "Columbus", "Sacramento"]
capitals_dictionary = dict(zip(keys, values)) # zip converts two lists in key-value pairs
names = ("Tony", "Jack", "Tom", "Merlyn")
companies = ("Google", "Apple", "Facebook", "Microsoft")
zipped = zip(names, companies)
print(list(zipped)) # return each element from two list as pairs in as new list
[ ('Tony', 'Google'), ('Jack', 'Apple'), ('Tom', 'Facebook'), ('Merlyn','Microsoft') ]
for (a,b) in zipped:
print(a,b)
The zip function also allows to iterate through multiple lists in parallel and access the corresponding element.
Python allows to use decorators in a simpler way using the @ symbol. In the below modified say_whee() method, we use @msg_decorator which is just an easier way of saying say_whee = msg_decorator(say_whee).
x_coordinates = [34, 56, 67, 89]
y_coordinates = [78, 76, 43, 12]
for x, y in zip(x_coordinates, y_coordinates):
print(x, y)
Functions
Functions are always defined first and then called in Python. It is recommended to have 2 blank lines after the function definition. The immutable arguments (int, string) passed to the function are passed by value, while mutable values (list, set etc) are passed by reference in python.
def greet_user(first_name, last_name, location):
print(f"Hello {first_name} {last_name} ! How is it in {location}")
The actual arguments which are actually passed to the function have below types.
Positional Argument: Positional Parameters are passed in the order of their definition in the function.
greet_user("John", "Smith", "Canada")
Keyword Argument: Keyword arguments are passed by keyword or name of argument in any order. The positional arguments and keyword arguments can be mixed within the function call. But keyword arguments should always come after positional arguments.
greet_user(location="London", last_name="Mayer", first_name="John")
greet_user("John", location="London", last_name="Mayer")
Default Argument: Set default value to the arguments of the functions so that those values can be skipped while calling the function.
def greet_user(first_name, last_name, location='USA'):
print(f"Hello {first_name} {last_name} ! How is it in {location}")
greet_user("John", "Smith")
Variable Length Argument: The number of arguments passed in a function is not fixed for variable length arguments. A '*' is added before the argument name in function definition. The variable length argument is of type tuple in the function.
def sum(*a):
r = 0
for i in a:
r += i
sum(4, 7, 9, 2)
Keyword variable length arguments is same as Variable length arguments, but it allows keyword arguments to be included in variable length. It is indicated by adding '**' before the argument name.
def person(name, **keyword_args):
print(name)
for i, j in keyword_args.items():
print(i, j)
person('Jack', age=56, city='London', contact=441711231233)
If there is no return statement in a function, then by default all the functions in python return None. A function returning None in Python is similar to void return in Java. Function can return multiple values instead of a single value as below.
Python allows maximum recursion depth of 1000. The sys.getrecursionlimit() function returns this recursion limit of python. The sys.setrecursionlimit(9999) allows to override the default value and set a custom recursion limit.
def count(l):
odd, even = 0, 0
for i in l:
if i%2 == 0:
even += 1
else:
odd += 1
return odd, even
lst = [3,5,7,8]
even, odd = count(lst)
Python allows maximum recursion depth of 1000. The sys.getrecursionlimit() function returns this recursion limit of python. The sys.setrecursionlimit(9999) allows to override the default value and set a custom recursion limit.
First-class functions
In Python, functions are first-class objects. This means that functions can be passed around and used as arguments, just like any other object. Also functions can be returned as values.
def say_hello(name):
return f"Hello {name}"
def be_awesome(name):
return f"Yo {name}, together we are the awesomest!"
def greet_bob(greeter_func):
return greeter_func("Bob")
greet_bob(say_hello)
greet_bob(be_awesome)
Inner Functions
Python allows to define a functions inside other functions. Such functions are called inner functions with below example. The inner functions are not defined until the parent function is called. They are locally scoped and only exist inside the parent() function.
def parent():
print("Parent function")
def first_child():
print("first child function")
def second_child():
print("second child function")
second_child()
first_child()
Function Annotations
Function annotations is a Python 3 feature that allows to add arbitrary metadata to function arguments and return value. The annotations only provides a nice syntactic support for associating metadata without any semantics (meaning) and is totally optional. In the below example function sum() which takes 3 arguments, a, b and c. The first argument a is not annotated, the second argument b is annotated with the string ‘annotating b’, and the third argument c is annotated with type int. The return value is annotated with the type float. Note the "->" syntax for annotating the return value. The annotations have no impact whatsoever on the execution of the function. Annotations have no standard meaning or semantics and is mainly used for documentation.
def sum(a, b: 'annotating b', c: int) -> float:
print(a + b + c)
Modules
A module is a python file which contains all the related functions and classes. The module can be imported into another python file in order to execute functions. The "import module-name" statement imports entire module in the python file. It also requires to specify the "module-name.function()" to call the function.
In the below example we have the Calc.py file which has add() function. Then another demo.py file is trying to call the add() function from Calc.py.
import Calc a = 4 b = 7 c = Calc.add(a, b)
In order to import only selected functions in python module we use "from module-name import function". It allows to call the function() directly without using the prefix 'modulename.'.
from math import sort, pow pow(3,2)
All the related modules are organized into a package which is a directory containing module files. Package is the container for multiple modules. A special file called "__init__.py" is added to the package directory to make the directory a package. When the python interpreter sees the "__init__.py" file in a directory then it treats the directory as a package. The modules within the package can be imported using "import package-name.module-name" or "from package-name.module-name import function".
Python has many standard built-in modules for many general functionality. The complete list is available in python 3 module index documentation. These modules can be imported directly without specifying the package name. The built in modules are located in python 3.x directory (library root) within the base python directory.
__name__ global variable
In Python Introspection is the ability of an object to know about its own attributes at runtime. For instance, a function knows its own name and documentation. The __name__ is once such a special global variable in python whose value depends on the place it is fetched. The value of __name__ is __main__ in the file were the python code is being executed. On other hand, its value is module name when its printed in another module imported in the main execution file. The value of the __name__ variable changes as per the place its being used. When running the python file as a main code, and using __name__ then it returns __main__. When __name__ is fetched in a file imported as a module then it returns module name. When a module is imported in python, it executes all the statements in the file. To avoid executing main method in the module file when imported as a module, but to allow execution of main method when ran in standalone mode, the __name__ variable is checked for __main__ as below.
def main():
print("Hello")
if __name__ == "__main__"
main()
Global Keyword
Global variables defined outside the function are accessible to the function in python. If the name of global variable and local variable in the function is same, then local variable will always take precedence within the function. If a global variable is assigned a value within a function it is interpreted as local variable. Hence to modify a global variable from within the function, a global keyword is used before the global variable name to explicitly specify access to global variable.
a = 90
def test():
global a
a = 34
print("value of a is ", a)
test()
print("Outside test value of a is ", a)
Also globals() provides access to all the global variables. To access a particular global variable 'a', we use globals()['a'].
a = 10
def test():
globals()['a'] = 15
Lambda Functions
Functions are objects in Python and they can be passed as parameters into other functions.
function = lambda parameters: body f = lambda a,b : a+b numbers = [2,3,7,9,6,4,5,8] # filter() example even_nums = list(filter(lambda n: n%2 == 0, numbers)) # map() example squares = list(map(lambda n: n*n, numbers)) # reduce() example from functools import reduce sum = reduce(lambda alb: a+b, numbers)
Iterators
The for loop uses iterators behind the scenes for looping items.
The iter() is the function which converts a list to an iterator, which is used to iterate the list one value at a time. The iterator has __next__() method which gives the current value and points to the next value. The iterator preserves the state of the last value returned by it.
numbers = [ 2, 15, 8, 6, 3, 4]
iterator = iter(numbers)
# Both below method prints the current value in the iterator and points to the next value
print(iterator.__next__())
print(next(iterator))
Example of iterator in a while loop.
def loop(iterable):
oIter = iterable.__iter__()
while True:
try:
print oIter.next()
except StopIteration:
break
loop([1,2,3])
A custom iterator can be created by implementing the __next__() and __iter__() methods.
class NumGenerator
def __init__(self, start, limit):
self.num = start
self.limit = limit
def __iter__():
return self
def __next__():
if self.num <= self.limit:
val = self.num
self.num += 1
return val
else:
raise StopIteration
generator = NumGenerator(1, 50)
for i in generator
print(i)
Generator
Generators are iterators, a kind of iterable we can only iterate over once. Generators do not store all the values in memory, they generate the values on the fly. The yield keyword is used to produce a sequence of values. It is used to iterate over a sequence without storing the entire sequence in memory. If the body of a def contains yield, the function automatically becomes a generator function.
def numgenerator()
yield 1
yield 2
yield 3
yield 4
yield 5
gen = numgenerator()
print(gen.__next__())
# Below for loop prints 2 to 5, as 1 is printed by above print statement
for i in gen
print(i)
Exceptions
In python, Exception is a generic error which includes all errors/exceptions. The try statement specifies exception handlers and/or cleanup code for a group of statements as below.
try:
age = int(input('Age: '))
income = 20000
average = income / age
print(f"Age is {age}")
except ZeroDivisionError:
print('Age cannot be zero')
except ValueError:
print('Invalid value')
except Exception as e:
print("Something went wrong...", e)
finally:
print("execution completed")
The try statement also supports else block after the except block, which is executed when there is no exception.
value = '9X'
try:
print(int(value))
except:
print('Conversion failed !')
else:
print('Conversion successful !')
With Statement
The with statement wraps the execution of a block with methods defined by a context manager. This allows common try…except…finally usage patterns to be encapsulated for convenient reuse. In the below example, the with statement automatically closes the file after the nested block of code, no matter how the block exits. If an exception occurs before the end of the block, it will close the file before the exception is caught by an outer exception handler. If the nested block were to contain a return statement, or a continue or break statement, the with statement would automatically close the file in those cases as well.
with open('output.txt', 'w') as f:
f.write('Hi there!')
Classes
Everything in Python is an Object. Classes are used to define new type or objects. The class can have methods in its body and they can also have attributes which can be set anywhere in the program.
# In Python we use camel case naming to name the class
class Point:
def draw(self):
print("draw")
point1 = Point()
point1.x = 10 # Creates an attribute x in point1 object and assigns the value 10
class Person:
country = "USA"
def __init__(self, name, age):
self.name = name # self is reference to current object. It adds new attribute 'name' and assigns the parameter value
self.age = age
def greeting(self):
print(f"hello, {self.name}")
def compare(self, other):
if self.age == other.age:
return true
else:
return false
employee = Person("John", 90)
# Two ways to call greeting() method
Person.greeting(employee)
# Here the object on which the method is called, internally passes itself as an agrument to self.
employee.greeting()
# Update the attribute of the object externally
employee.name = "Tim"
employee.greeting()
manager = Person("Michael", 90)
# Compare takes two parameters, who is calling it and whom to compare with.
if manager.compare(employee):
print("Employee and Manager have same age")
# class variables are accessed same as instance variables.
print(manager.name, manager.age, manager.country)
# update class variable
Person.country = "Canada"
Init and New Methods
In Python, __init__() method is responsible for instantiating the class instance. It acts as the constructor of the class and takes parameters to set attribute values. The __init__() constructor is optional for a class. The __new__ method is similar to the __init__ method, and is called when the class is ready to instantiate itself. The major difference between these two methods is that __new__ handles object creation and __init__ handles object initialization. The __new__() method is defined as a static method in the base class and it needs to pass a class (cls) parameter. The class (cls) parameter represents the class that needs to be instantiated, and this parameter is provided automatically by python parser at instantiation time. The __new__ method is called first when an object is created and __init__ method is later called to initialize the object. If both __init__ method and __new__ method exists in the class, then the __new__ method is executed first and decides whether to use __init__ method or not. The reason being that the new() method can call other class constructors or simply return other objects as instances of this class.
The self parameter represents the current (object) instance of the class. The self keyword allows to access the attributes and methods of the class in the python. It binds the attributes with the given arguments.
class Employee(object):
def __init__(self, name, salary):
self.name = name
self.salary = salary
def __new__(cls, name, salary):
if 0 < salary < 10000:
return object.__new__(cls)
else:
return None
def __str__(self):
return '{0}({1})'.format(self.__class__.__name__, self.__dict__)
emp = Employee("James", 4500)
print(emp)
Two types of variables in Python class, an instance variable and a class (static) variable which is common for all the objects. The variables defined within __init__() are instance variables, while the variables defined outside __init__() in the class are called class variables. Python has namespace which is an area where an object/variable is created and stored. Class variables are stored in class namespace, while instance variables are stored in Object/instance namespace.
There are two types of methods in Python:
Instance methods: The methods which take the "self" parameter are called instance methods. Instance methods have two types, accessor methods which fetch values and mutator methods which modify values.
Class methods: Class methods are common to all objects and are used to work with class variables. All class methods have the parameter "cls" in their methods.
class Person:
country = "USA"
@classmethod
def country(cls):
return cls.country
p1 = Person()
print(p1.country())
print(Person.country())
To call the class method we need to pass cls parameter which can be avoided by adding @classmethod decorator to the class method.
Static methods: The method which neither uses the instance variables nor class variables and provides independent functionality is called a static method. Mainly it is used for utility methods. For static method we need to use the @staticmethod decorator.
class Person:
def info():
print("Information about the class Person")
Person.info()
Meta-Classes
In Python the object.__class__ designates the name of the class for the object. From Python 3 an object’s type and its class is referred interchangeably, as type(object) is the same as object.__class__. Since everything is an object in Python, all the classes also have a type, which is the type class itself. The type class is a metaclass, of which all the classes are instances.
The built-in type() function enables to create new class dynamically. It takes the name of the class, tuple of base classes which the class inherits and namespace dictionary containing the definitions for class body. When all these parameters are passed the type() function dynamically defines a class.
# create new class Person dynamically. Here attr_val can also be assigned to an external function name
Person = type('Person', (), { 'attr': 100, 'attr_val': lambda x : x.attr })
p = Person()
print(p.attr_val()) # prints value 100 of attr
# create new class Employee extending Person class dynamically
Employee = type('Employee', (Person,), dict(attr=100))
e = Employee()
print(e.attr) # prints 100
print(e.__class__) # prints class '__main__.Employee'
print(e.__class__.__bases__) # prints tuple with single element, class '__main__.Person'
A class in Python can be instantiated using the expression e.g. Person() which creates a new instance of class Person. When the interpreter encounters Person(), it first calls the __call__() method of Person’s parent class. Since Person is a standard new-style class, its parent class is the type metaclass, so type’s __call__() method is invoked. This __call__() method in turn invokes the __new__() and __init__() methods. If Person does not define __new__() and __init__(), default methods are inherited from Person’s ancestry. But if Person does define these methods, they override those from the ancestry, which allows for customized behavior when instantiating Person.
def new(cls):
x = object.__new__(cls)
x.attr = 500
return x
# modify instantiation behavior of class Person by initializing an attribute attr to 500
Person.__new__ = new
g = Person()
print(g.attr) # prints 500
Python does not allow to reassign the __new__() method of the type metaclass. To customize the instantiation of the class, a custom meta class can be created by extending the type meta class and overriding the __new__() method. While defining a new class we specify that its metaclass is a custom metaclass using the metaclass keyword in the class definition, rather than the standard metaclass type. Such custom meta-class serve as a template for creating classes and referred to as class factories.
class Meta(type):
def __new__(cls, name, bases, dct):
x = super().__new__(cls, name, bases, dct)
x.attr = 100
return x
def __init__(cls, name, bases, dct):
cls.attr = 100
class Foo(metaclass=Meta):
pass
print(Foo.attr)
Inner Classes
Python also allows to have inner classes as shown in below example. We can create object of inner class inside the outer class or outside the outer class provided the outer class name is used to call it.
class Person:
def __init__(self, name, age):
self.name = name
self.age = age
self.address = self.Address()
def show(self):
print(self.name, self.age)
self.address.show()
class Address:
def __init__(self):
self.street = "Main Street"
self.city = "Boston"
self.country = "USA"
def show(self):
print(self.street, self.city, self.country)
p1 = Person("Jimmy", 2)
p1.show()
# access attributes of inner class
print(p1.address.street)
new_address = Person.Address()
Inheritance
Python allows to inherit all the methods from the parent class. It also allows multiple inheritance. Every class in Python is derived from the object class which is the base type in Python.
class Mammal:
def __init__(self):
print("Mammal Init")
def breathe(self):
print("breathe oxygen")
class Fish:
def __init__(self):
print("Fish Init")
def swim(self):
print("swims")
# Python does not like empty class, so a 'pass' line is added to let python know to pass this line.
class Dog(Mammal):
pass
class Cat(Mammal)
def __init__(self):
super().__init__()
print("Cat Init")
def runs(self):
print("runs 30mph")
class Whale(Mammal,Fish)
def __init__(self):
super().__init__() # By default calls the Mammal's init method
print("Cat Init")
def color(self):
print("color of whale is blue")
Python always executes the __init__() method of the object's class. If it cannot find the __init__() method in the sub class then it will call the init method of the super class. To call explicitly call init() method of the super class from subclass or any other methods in super class, the super() keyword is used. As python supports multiple inheritance, when a sub class inheriting from multiple super classes calls super().__init__() method from its own __init__() method then by default it calls the init method of the first Super class mentioned in the inheritance list. Python has a Method resolution order which starts from left to right of the super classes in multiple inheritance. Hence the first super class mentioned in the multiple inheritance is called for init() or any other method using super().method_name().
Method Resolution Order
Method Resolution Order (MRO) determines the order in which base class methods should be inherited in the case of multiple inheritance. It defines the order in which the base classes are searched when executing a method. First the specified method or attribute is searched within the current class. If not found, the search continues into parent classes in depth-first, left-right fashion, in the order specified while inheriting the classes, without searching the same class twice. MRO ensures that a class always appears before its parents. In case of multiple parents, the order is the same as tuples of base classes. In the below example of diamond inheritance Python follows a depth-first lookup order i.e. Class D -> Class B -> Class C -> Class A, which ends up calling the method from class B.
class A:
def hello(self):
print(" In class A")
class B(A):
def hello(self):
print(" In class B")
class C(A):
def hello(self):
print("In class C")
# multiple inheritence
class D(B, C):
pass
r = D()
r.hello()
MRO of a class can be viewed as the __mro__ attribute or the mro() method. The former returns a tuple while the latter returns a list.
class X:
pass
class Y:
pass
class Z:
pass
class A(X, Y):
pass
class B(Y, Z):
pass
class M(B, A, Z):
pass
print(M.mro())
Polymorphism
Duck Typing:
class PyCharm
def execute(self)
print("Compiling")
print("Running")
class Sublime
def execute(self)
print("Spell Check")
print("Compiling")
print("Running")
class Computer
# Here ide variable takes any type as long as it has the execute() method
def code(self, ide)
ide.execute()
comp = Computer()
pycharm = PyCharm()
comp.code(pycharm)
sublime = Sublime()
comp.code(sublime)
Operator Overloading:
class Business
def __init__(self, expense, sales):
self.expense = expense
self.sales = sales
def __add__(self, other):
expense = self.expense + other.expense
sales = self.sales + other.sales
business_obj = Business(expense, sales)
return business_obj
def __str__(self):
return '{} {}'.format(self.sales, self.expense)
b1 = Business(1300, 2100)
b2 = Business(9800, 7300)
# Python internally converts the + operator expression to method call Business.__add__(b1, b2)
b3 = b1 + b2
Similar as the '+' is converted to predefined __add__() method call, other operators are converted to python methods as mentioned . Also python invokes __str__() method behind the scene when we try to print the value of any object.
# Internally python calls print(b1.__str__()) print(b1)
Method Overloading: Python does not support method overloading were we can have two methods with same name but different (no of) arguments. But python does allow to set default values to method parameters making them optional while invoking the method as below.
class Sample
def sum(self, a=None, b=None, c=None):
s = 0
if a != None and b != None and c != None:
s = a + b + c
elif a != None and b != None:
s = a + b
else:
s = a
return s
s1 = Sample()
print(s1.sum(4,7,9))
print(s1.sum(2,7))
Method Overriding: Python supports Method Overriding, and invokes the method in the current class rather than the super class method with same name.
class A
def show(self)
print("In Class A")
class B(A)
def show(self)
print("In Class B")
b1 = B()
b1.show()
Abstract Class
By default python does not support abstract classes. The ABC (Abstract Base Classes) module is used to implement abstract class in python.
from abc import ABC, abstractmethod
class Computer(ABC): # The abstract class with abstract methods should inherit from class ABC
@abstractmethod # Abstract methods are annotated with abstractmethod annotation
def process(self):
pass
class Laptop(Computer):
def process(self):
print("Running")
com = Computer()
com.process() # Gives error as process() is abstract method in Computer class
lap = Laptop()
lap.process()
Decorators
Decorators allow to add extra features to the existing functions without modifying the actual function. In other words the behavior of the existing function is changed using decorator by adding a new function which invokes the original function. It modifies the behavior of a function without permanently modifying it, by wrapping the original function with another function. Python allows to pass a function as a parameter to another function.
def msg_decorator(function):
def wrapper():
print("Hello")
function()
print("Welcome to Python Tutorial !")
return wrapper
def say_whee():
print("Whee !")
say_whee = msg_decorator(say_whee)
say_whee()
Python allows to use decorators in a simpler way using the @ symbol. In the below modified say_whee() method, we use @msg_decorator which is just an easier way of saying say_whee = msg_decorator(say_whee).
@msg_decorator
def say_whee():
print("Whee !")
The *args and **kwargs are used in the inner wrapper function to support it to accept an arbitrary number of positional and keyword arguments. Similarly the wrapper function should support return value if the decorated function to return values from decorated functions. Further the name of the original function (using .__name__) after being decorated could be fixed by using the @functools.wraps decorator.
import functools
def decorator(func):
@functools.wraps(func)
def wrapper_decorator(*args, **kwargs):
# Do something before
value = func(*args, **kwargs)
# Do something after
return value
return wrapper_decorator
Decorating Classes
The methods of a class can be decorated similar as the functions. Some commonly used decorators that are built into Python are @classmethod, @staticmethod, and @property. The @classmethod and @staticmethod decorators are used to define methods inside a class namespace that’s not connected to a particular instance of that class. The @property decorator is used to customize getters and setters for class attributes.
class Circle:
def __init__(self, radius):
self._radius = radius
@property
def radius(self):
return self._radius
# define a setter method
@radius.setter
def radius(self, value):
if value >= 0:
self._radius = value
else:
raise ValueError("Radius must be positive")
@property
def area(self):
return self.pi() * self.radius**2
def cylinder_volume(self, height):
return self.area * height
@classmethod
def unit_circle(cls):
return cls(1)
@staticmethod
def pi():
return 3.1415926535
Decorators can be used on the entire class as a simpler alternative to metaclasses. The only difference is that the decorator will receive a class and not a function as an argument. The decorator class takes a function as an argument in its .__init__() method and stores a reference to the function. The class instance is also callable, by implementing the special .__call__() method, so that it can stand in for the decorated function. The .__call__() method will be called instead of the decorated function. The functools.update_wrapper() function is used instead of @functools.wraps.
class CountCalls:
def __init__(self, func):
functools.update_wrapper(self, func)
self.func = func
self.num_calls = 0
def __call__(self, *args, **kwargs):
self.num_calls += 1
print(f"Call {self.num_calls} of {self.func.__name__!r}")
return self.func(*args, **kwargs)
@CountCalls
def say_whee():
print("Whee!")
File Handling
Files can be access in python using open() function. It could be opened in read (r), write (w) or append (a) mode using character format by default.
file = open("file.txt", a)
if file.readable():
print(file.read()) # read entire file
print(file.readline()) # read single line
file.write("something")
file.close()
There are additional modes named read (rb), write (wb) or append (ab) to access/write binary files. Refer to the complete list of file access modes for details. Below example copies image data from one file to another file.f1 = open('image1.jpg', 'rb')
f2 = open('image2.jpg', 'wb')
for data in f1:
f2.write(data)
# The glob() method allows to search for files or directories in the current path.
for file in path.glob('*.py')
print(file)
Pathlib module provides an object oreinted filesystem paths, i.e. it provides classes to work with files and directories.path = Path("tempdirectory")
path.exists()
path.mkdir()
path.rmdir()
Multi Threading
from time import sleep
from threading import *
class Greeting(Thread):
# run() is a method in Thread class which needs to be overridden to implement thread.
def run(self):
for i in range(500)
print("Hello")
sleep(1) # sleep takes number of seconds for the execution to be suspended
class Message(Thread):
def run(self):
for i in range(500)
print("Welcome")
sleep(1)
g = Greeting()
m = Message()
# The start() method of Thread class internally invokes the run() method
g.start()
m.start()
# Ask main thread to wait until thread m and thread g completes
g.join()
m.join()
print("Thread program completed")
PIP and PyPI
Pip is the standard package environment system to install and manage software package in Python. It allows you to install and manage additional packages that are not part of the Python standard library. Pip has been included with the Python installer since versions 3.4 for Python 3, as package management is important for application development.
Python has a very active community that contributes an even bigger set of packages than the Python standard library, which helps with our development needs. These packages are published to the Python Package Index, also known as PyPI. PyPI hosts an extensive collection of packages that include development frameworks, tools, and libraries. For example PyPI hosts a very popular library to perform HTTP requests called requests. To install any package, the command "pip install <package>" is used. Pip then looks for the package in PyPI, calculates its dependencies, and finally installs the package. The package is installed in python 3 directory under site packages folder. The pip install command always installs the latest published version of a package.
$ pip install requests
The list command shows the packages installed in the environment.
$ pip list
To view the package metadata the show command in pip is used.
$ pip show requests
To Install the specific version of a package we use.
$ pip install "SomeProject==1.4"
To install greater than or equal to one version and less than another:
$ pip install "SomeProject>=1,<2"
To install a version that’s “compatible” with a certain version: 4
$ pip install "SomeProject~=1.4.2"
To upgrade an already installed package to the latest from PyPI.
$ pip install --upgrade SomeProject
The --upgrade option upgrades all specified packages to the newest available version. The --force-reinstall option reinstalls all packages even if they are already up-to-date. The --ignore-installed option ignores whether the package and its deps are already installed, overwriting installed files. The --no-deps option doesn't install package dependencies.
$ pip install --upgrade --force-reinstall --ignore-installed --no-deps <package>
We can create a specification of the dependencies and versions which would be used to develop and test the application. Requirement files allow to specify exactly which packages and versions should be installed. The pip freeze command outputs all the packages installed and their versions to the standard output. This output from freeze command in requirements format can be redirected to generate a requirements file.
$ pip freeze > requirements.txt
The requirement.txt would contain out in below format.
scikit-learn==0.22.1 matplotlib==3.1.3 numpy==1.11.2
To later replicate the environment in another system, we can run pip install specifying the requirements file (or any other text file name) using the -r switch as below.
$ pip install -r requirements.txt
Pip search command allows to search in the PyPI for any libraries using multiple search keywords.
$ pip search requests oauth
Package can be uninstalled using the uninstall command.
$ pip uninstall request
Pip by default gets the latest version of python packages, including the sub-dependencies which could cause issues especially in production if the version is non-backward compatible. Typically this is resolved by executing the 'pip freeze' command which pins all dependencies and freezes everything in development to requirement.txt file. Although now each specified versions of third-party packages including its sub-dependencies needs to be manually updated ensuring inter compatibility, which can become cumbersome to manage.
Virtual Environment
Virtual environment is an indispensable part of Python programming. A virtual environment is an isolated container containing all the software dependencies for a given project. It has its own Python executable and third-party package storage. It is important because by default software (packages) such as Python, NumPy and Django are installed in the same directory. Python stores all the system packages in a child directory of the path from "sys.prefix" and all the third party packages are placed in one of the directories pointed by "site.getsitepackages()". This causes a problem when we want to work on multiple projects on the same computer. If a project uses one version of NumPy while some other project uses a different NumPy version, then the virtual environment provides an isolated environment to separate the two project setups effectively. There are no limits to the number of virtual environments we can have since they’re just directories containing a few scripts. The venv module which is part of standard library in Python 3 enables to create the lightweight virtual environments mentioned before.
The virtualenv tool can be installed with pip using the below command.
$ pip install virtualenv
Create a new virtual environment inside a new directory 'virtual-env'. The below command works only for Python 3.
$ mkdir virtual-env && cd virtual-env
$ python3 -m venv envname
Alternatively we can use the below command to create and start virtual environment named 'envname', using the default python version. It creates the virtual environment within the current directory.
$ virtualenv envname
Create and start virtual environment using the specified python version.
$ virtualenv envname -p python3
$ virtualenv envname -p /usr/local/bin/python3
The above command creates a env directory which contains bin directory with all python binaries, include directory with python packages and finally the lib directory were all third party dependencies are installed in site-packages. The activate scripts in the bin directory allow to to set up the shell to use the environment’s Python executable and its site-packages by default. In order to use this environment's packages/resources in isolation, we need to “activate” it using the below command.
$ source envname/bin/activate
For windows the command is '.\Scripts\activate'. With the above command the prompt is prefixed with the name of the environment (envname) indicating that the 'envname' environment is active. To exit from the environment use the deactivate command.
(envname) $ deactivate
On activation of the environment, the virtual environment’s bin directory is the first directory searched when running an executable on the command line. Thus, the shell uses our virtual environment’s instance of Python instead of the system-wide version.
The virtualenvwrapper tool is a wrapper scripts around the main virtualenv tool, which helps in organizing all the virtual environments in one location. It also provides methods to easily create, delete, and copy environments, and switch between multiple environments.
Pipenv
Pipenv is a packaging tool which simplifies the dependency management in Python-based projects. It brings together Pip, Pipfile and Virtualenv to provide a straightforward and powerful command line tool. Pipenv has virtual environment management built in which makes it a single tool for both package and environment management. The virtual environment for the project is created and managed by Pipenv when packages are installed via Pipenv’s command-line interface. Dependencies are tracked and locked, with development and production dependencies managed separately. Pipenv is installed using below pip command
$ pip install pipenv
The pipenv install command is used to install (all or specific) packages within the project. It also creates two files, Pipfile and Pipfile.lock, and a new virtual environment in the project directory. If no python version is specified, it uses the default version of Python. The Pipfile contains dependency information about the project and is used to track the project dependencies. The Pipfile supercedes the requirements.txt file that is typically used in Python projects. The pipenv install command installs all the packages specified within the Pipfile.
$ pipenv install
Specific package can be installed or removed using the pipenv's install and uninstall commands as below
$ pipenv install requests
$ pipenv uninstall requests
Pipenv enables to keep two environments separate using the --dev flag and install pytest in dev environment.
$ pipenv install pytest --dev
To completely wipe all the installed packages from your virtual environment, we use below command.
$ pipenv uninstall --all
The package name along with its version and its dependencies, can be frozen by updating the Pipfile.lock. This is done using the below lock command.
$ pipenv lock
In order to activate or create the virtual environment associated with the Python project below shell command for pipenv is used.
$ pipenv shell
To invoke shell commands in the virtual environment, without explicitly activating it first, the run command is used.
$ pipenv run <insert command here>
Pipenv's graph command displays a dependency graph to understand the top-level dependencies and their sub-dependencies.
$ pipenv graph
Pipenv supports the automatic loading of environmental variables when a .env file (containing key-value pairs) exists in the top-level directory. In such case, when the pipenv shell command opens the virtual environment, it loads the environmental variables from the file. The default behavior of Pipenv can be changed using some environmental variables for configuration.
Pipenv has many advance features such as specifying a package index, detection of security vulnerabilities, easily handling environment variables, and playing nicely with Windows. Some of the drawbacks of pipenv include, generation of many miscellaneous files in the project root directory, performance issues with dependency resolution and has complex commands/options. Also pipenv does not manage the scaffolding (internal structure) of the project unlike Poetry.
Poetry
Poetry is a modern tool which simplifies dependency management and packaging in Python. It manages project dependencies, creation and activation of virtualenv, build and publishing of packages, ensures package integrity and allows to convert python functions to command line programs. It also provides directory structure for the project including tests.
Install poetry using below pip command
$ pip install poetry
Below are some of the poetry commands to setup project, fetch dependencies and execute project.
It is recommended to configure poetry to create the project's virtual environment in .venv folder inside the project directory, before using poetry to create a project. This is very handy when using IDEs like VS Code and PyCharm as they immediately recognizes the .venv folder and pick up the correct interpreter.
$ poetry config virtualenvs.in-project true
Create a new project by creating a directory structure for the project.
$ poetry new poetry-demo
The above command creates the project's directory structure containing poet and test directories, and the pyproject.toml file. The .toml file defines the project metadata, all project build dependencies, development dependencies to perfom other actions like testing, building, documentation, etc and finally the build system. Poetry also automatically creates the virtual environment for the project, if it detects no virtual environment already associated with the project. The environment info command in poetry displays the path of the current virtual environment with other details.
$ poetry env info
The poetry init command creates a pyproject.toml file interactively by prompting to provide basic information about the package.
$ poetry init
The install command reads the pyproject.toml file from the current project, resolves all the dependencies, and installs them.
$ poetry install
During the installation, poetry automatically generates the poetry.lock file to track the exact version of the dependencies that have been install on the system. If the poetry.lock file is already present, it would install the exact version of packages defined in the lock file, instead of trying to install the latest one from PyPi. It is recommended to track poetry.lock along with pyproject.toml file in source control.
In order to get the latest versions of the dependencies and to update the poetry.lock file, you should use the update command
$ poetry update
The add command adds required packages to your pyproject.toml and installs them. The --dev flag is used to install a development dependency which is not directly related to the project.
$ poetry add pandas
$ poetry add flake8 --dev
The lock command locks (without installing) the dependencies specified in pyproject.toml.
$ poetry lock
The show command lists all of the available packages. The -t or --tree option lists all the dependencies as well as the sub-dependencies in tree format. The --latest option allows to check if there is any latest version available for the package dependency.
$ poetry show
$ poetry show --tree
$ poetry show --latest
The run command executes the given command inside the project's virtual environment. It can also execute one of the scripts defined in pyproject.toml.
$ poetry run pytest
$ poetry run python main.py
Poetry also provides a shell command that spawns a new shell directly inside the virtual environment. It enables to execute commands in virtual environment without using 'poetry run' in front of the command.
$ poetry shell
> pytest
The update command can be used to update all or specific package dependencies.
$ poetry update
$ poetry update pytest
The remove command is used to remove the package from the project. To remove a development package the -D or --dev option must be specified.
$ poetry remove requests
$ poetry remove -D pytest
The build command builds the source and wheels archives.
$ poetry build my_new_project
$ poetry build
The publish command deploys the package built previously to either a public or private repository. Specifying the build option allows to build the package before publishing. By default the publish command publishes the package in pipy public repository. To switch to a private repository -r option is used along with the name of the private repository configured using the config command.
$ poetry publish
$ poetry publish -r privrepo -u username -p password
The config command enables to configure a private repository name with the repository url.
$ poetry config repositories.privrepo https://private.repository
We can also store credentials for the private repository using the config command.
$ poetry config http-basic.privrepo username password
The credentials for pypi can also be configured using the previous command and replacing 'privrepo' by 'pypi' but it is now recommended to use API tokens to authenticate with pypi as below.
$ poetry config pypi-token.pypi my-token
No comments:
Post a Comment