Quick guide to Logging in different files using Python

This is for quick setup of a logging system . 


Step 1: import logging 


Logging is the build-in module for implementing logging in Python.

Step 2: Create a logger for yourself. 


logger = logging.getLogger(__name__)

Step 3: Set Logger level


logger.setLevel(logging.DEBUG)

Step 4: Create a handler. 


hdlr = logging.FileHandler(settings.BASE_LOG_FILE, mode='w')

Step 5: Create a formatter


formatter = logging.Formatter('%(asctime)s %(levelname)s %(message)s')

Step 6: Set formatter and handler


hdlr.setFormatter(formatter)

logger.addHandler(hdlr)



When and why to use Classes in Python

Note: All programming done in Python 2.7

Why to make a class?

When we need to use some common variables across other functions quite often . If we only one or 2 functions to write , we should not need to write a class. But if we are going to have 4-5 functions that need to use common objects between themselves, then its better to use classes.

What are class methods?

Class methods are methods that you can call on the class itself

A good example of classes . Lets have a BankAccount class for managing Bank Accounts and its related operations.


class BankAccount(object):
    #Class methods are methods that you can call on the class itself
    def __init__(self, balance=0):
        self.balance = balance
    def deposit(self, amount):
        self.balance += amount
    def withdraw(self, amount):
        self.balance -= amount

The __init__ function is automatically called when a new instance is created. We have two functions that add or subtract  money from account.

Ok....So lets create 2 accounts


a_account = BankAccount()
b_account = BankAccount()

Now lets deposit something in these accounts


>>> a_account.deposit(100)
>>> b_account.deposit(500)


Let's try to transfer some fund from B to A account. We need to add a "transfer" function to our class.


def transfer(self, other_account, amount):
    self.withdraw(amount)
    other_account.deposit(amount)


>>> b_account.transfer(a_account, 100)
>>> a_account.balance
200
>>> b_account.balance
400

As you can see, how we used classes to perform simple operations between two different accounts.

Customize looks of your object

We can have more control over our objects . Like if we print our object , what will it look.
Normally it looks like this.


>>> a_account
<__main__.BankAccount object at 0x0000000001FB3B38>

We want to show some other info when someone calls up the object itself at the time it is created. We can use the __repr__ function for this.

Lets add this function to our class .


def __repr__(self):
    return "Bank_Account Balance={}".format(self.balance)

One thing to note:

 if __repr__ is defined, and __str__ is not, the object will behave as though __str__=__repr__.

Find more info here 


>>> a_account = BankAccount(1000)
>>> a_account
Bank_Account Balance=1000

If we want to control how it will look when its printed , we should use the __str__ method.


def __str__(self):  # When the object is printed or str(object) is used
    return "Bank_Account is ${}".format(self.balance)


What is Super Class

You might have seen before the keyword being used inside classes. Super here helps us call the original base class __init__function . Whenever I have a method to call in the original baseclass.


class NamedBankAccount(BankAccount):
    def __init__(self, name, *args, **kwargs):
        self.name = name
        super(NamedBankAccount, self).__init__(*args, **kwargs)

Now we create an object of above class.


>>> a = NamedBankAccount('Arc', 1000)
>>> a.name
'Arc'
>>> a.balance
1000
>>> print a
Bank_Account is $1000



As you can see, the 'balance' was auto created by the __init__ of BankAccount class.

Note: In python 3, you can use super().__init__()   . This is much simpler.

Lets add a hello function in our base class and lets add that in our Child class.


class BankAccount(object):
    #Class methods are methods that you can call on the class itself
    def __init__(self, balance=0):
        self.balance = balance
    def deposit(self, amount):
        self.balance += amount
    def withdraw(self, amount):
        self.balance -= amount
    def transfer(self, other_account, amount):
        self.withdraw(amount)
        other_account.deposit(amount)
    def __repr__(self):
        return "Bank_Account Balance={}".format(self.balance)
    def __str__(self):  # When the object is printed or str(object) is used
        return "Bank_Account is ${}".format(self.balance)
    def hello(self):
        print 'Hi'



class NamedBankAccount(BankAccount):
    def __init__(self, name, *args, **kwargs):
        self.name = name
        super(NamedBankAccount, self).__init__(*args, **kwargs)
        super(NamedBankAccount, self).hello()


>>> a = NamedBankAccount(name='Arc')
Hi

I think this is sufficient proof enough that we can call ANY method using super. It just a convenience provided by python.

Note: Just remember that the same can be achieved simple like this, 


BankAccount.__init__(self, 234)

>>> a = NamedBankAccount(name='Arc')
>>> a
Bank_Account Balance=234


Generator functions. Why and when should we use them.

You probably have heard of them. But as a beginner python programmer, we tend to ignore it's values because we fail to find uses of it. When we have loops and itertools and other fancy toys offered, why should we bother to understand them. If you want to take your programming  or understanding of Python to the next level, you must understand and use generators more and more.

Before we start, I want to make a simple program that prints square of (hence generates and delivers) 5 numbers from 1 to 5. So what is the first idea that pops up ? Let's use a for loop.


def square(N):
    for i in range(1, N+1):
        print i**2

>>>square(5)
1
4
9
16
25

Let's say, you need all the values generated for some other calculation. I know, above example is too lame . But imagine after some complicated calculations, you have generated some result.
So you probably will return all values together in an array. And to that, the first thought in our minds is, using a LIST. Let's use it. 

Now our function will look like this.

>>> def my_function(N):

    new_list = []

    for i in range(1, N+1):

        new_list.append(i**2)

    return new_list




>>> my_function(5)
[1, 4, 9, 16, 25]


All this looks easy. You probably are thinking...If generators are going to ruin this idea, I don't need it. Think again.
  1.  This LIST business will soon start becoming a nuisance as the  number of such calculations and variables increase (believe  me...thinking of new names for variables... SUCK).
  2.  What if you don't know your requirement. How many results  do you want.?
  3.  You had to create an array and it will take memory space  which becomes bigger with length of the list and size of each  element.
  4. Once you are dealing with very large list (imagine a million), generating a list of 1 million (let's assume integers) will take around 300 mb in memory.
A generator will make it all...Simpler. Here's the same example with generator function.


def square(N):
    for i in range(1, N+1):
        yield i**2

gen = square(5)

print gen.next()
print gen.next()
print gen.next()
print gen.next()
print gen.next()
print gen.next()

Result:

1
4
9
16
25

Traceback (most recent call last):

  File "D:\mydrive\Desktop\PyScripts\generators.py", line 14, in <module>
    print gen.next()
StopIteration
>>>

You can guess the reasons for the last traceback. We got out of fuel. In order to avoid this, you can always use generator object to iterate.


def square(N):
    for i in range(1, N+1):

        yield i**2

gen_object = square(5)

for result in gen_object:

    print result

Result:

1
4
9
16
25


More fun with generators:

You can also make generator functions from list comprehensions. Use parenthesis instead of square bracket to make a gen function.


l = (x**2 for x in range(1,6))  

print l
for value in l:
    print value


Result:

<generator object <genexpr> at 0x0000000002D50F78>
1
4
9
16
25

Python: Running executables using subprocess module

There are many ways to run a bat/exe file but the most widely used is the subprocess module.
And I would like to clearly show, how exactly to do that. When should you use what arguments.
I am going to specifically talk about the check_output module .

Note : This function is used when we want the output of the bat file collected inside a variable.

Condition 1: The exe file is in the same location as of the python script that wants to run it.

Explanation: 
You are in a Folder > MyPrograms  . And the bat/exe file lies inside this folder. (Refer picture below)

Single folder with bat and python files

Content of the hello.bat file


@echo off
echo "This is a test bat"


My script contents:


import subprocess
result = subprocess.check_output('hello.bat')
print result


Now if you run it , the result will be a:

"This is a test bat"

This is easy. Let's look at another condition.



Condition 2: The exe file is in a different location from the python script that wants to run it.

Two diff folders with python script and bat file

No change in contents of hello.bat file. Some change in myscript.py file.


import subprocess
result = subprocess.check_output('hello.bat', shell=True, cwd=r'D:\inchowar\Desktop\temp')
print result


The trick here is, cwd (current working directly) argument helps you automatically switch switch to that location specified before it tries to find the bat file . 

Otherwise you would have to do it yourself . Change directory before you run the bat file using os.chdir(r'D:\inchowar\Desktop\temp'), and then run the bat file, Obviously, this becomes tedious.

Note: When you are using the cwd argument, the bat file should not have a path before it. Only the name of bat file i.e hello.bat. NOT c:\Temp\hello.bat

As to when to use shell=True, I think this article has much better explanation than I can provide here.

Thanks for reading.

Regular Expressions with Python: Look ahead and Look behind

These two important tools have been bugging me every time I come up with a situation. Because I never remember this one. Blame my memory.
So it's time I cement this idea somewhere . And no better place than a blog page.

LOOK AHEAD

LOOK AHEAD means , I want to see what lies ahead first, and then decide, to do something now.


Syntax of regex pattern:  pattern1(?=pattern2)


Where Pattern1 is the pattern for the part that we ACTUALLY want to capture. Pattern2 is the pattern which needs to be found as MANDATORY. Logically speaking, 

IF PATTERN2 is FOUND, then print/get/capture/show PATTERN1


Example: Let there be a string "Hello World"

Aim: I want to find 'Hello' ...ONLY IF its followed by 'World'



re.search(r'\w+(?= World)', 'Hello World').group()

Result : >> 'Hello'

Explanation:

  •  The \w+ is Pattern1
  •  The 'World' is Pattern2
  •  r'\w+(?= World)' means, Find anything which fits \w+ IF it is followed by 'World'

Rules:
  • The Pattern2 needs to be in parenthesis.
  • If pattern2 itself has parenthesis, then they need to be bracketed. i.e [ ( ] or [ ) ]

LOOK BEHIND


Its just the opposite of the above. It means , I want to see what lies behind me , and then decide to do something.



Syntax of regex pattern:  (?<=pattern1)pattern2


Where pattern2 is the pattern for the part that we ACTUALLY want to capture. Pattern1 is the pattern which needs to found as MANDATORY.


IF PATTERN1 is FOUND, then print/get/capture/show PATTERN2


Example : We will take the same example. 'Hello World'

Aim : I want to find 'World' ONLY if it is preceded by 'Hello ' (The space here also counts)

>>> re.search(r'(?<=Hello )\w+', 'Hello World').group()

Result : 'World'

Rules: 
  • Pattern1 needs to be in parenthesis
  • If pattern2 itself has parenthesis, then they need to be bracketed. i.e [ ( ] or [ ) ]


LOOK AHEAD & LOOK BEHIND COMBINED

Consume only if it is surrounded by the things we want.



Syntax of regex pattern: (?<=pattern_behind)pattern_middle(?=pattern_ahead)


Where pattern_middle is the pattern for the part that we ACTUALLY want to capture. Pattern_behind and Pattern_ahead are patterns which need to be found as MANDATORY.


IF PATTERN_AHEAD AND IF PATTERN_BEHIND are BOTH found, consume PATTERN_MIDDLE


Example: We will take a new example . 'Hello My World'

Aim: I want to find any word that occurs in between Hello & World


>>> re.search('(?<=Hello )\w+(?= World)', 'Hello My World').group()

Result : 'My'

More examples:


Problem : Remove all special symbols with a space,  that come in between two alphanumeric character.
Target string'This$#is% Matrix#  %!'


re.sub(r'(?<=[a-zA-Z0-9])[$#@%^\s]+(?=[a-zA-Z0-9])', ' ', 'This$#is% Matrix#  %!')

Explanation: For alphanumeric character, we used [a-zA-Z0-9]

To find special characters between them , we use a combo of look ahead and look behind.

Kerala : How to plan for a lifetime's experience

Kovalam Beach
Kovalam Beach

I have been planning this for months and it finally came along in the best time. A full stretch Kerala trip with Kanyakumari as an added bonus. I will not tell you what to do, but HOW to do it. How to plan your dream Kerala Trip. 

Caution: Mine was too comprehensive. Needs high energy and will be quite tiring. 

I chose the monsoon theme..I wanted to be in the middle of the action. This is the only place in India which offers monsoon as a tourism opportunity. I was lucky. I expected super heavy rains. Instead. I ended up right in between the rainy season and summer which was Perfect.

Advantages in travelling in mid August

  • Rainfall almost over.
  • Waterfalls at their fullest.
  • Greenery all over the state.
  • Backwaters, beach sides and rivers all very clean.
  • Abundant coconuts to enjoy.
  • Hotel rates brought down.
  • Less crowd everywhere.
  • Festive Preparation going on.
  • Shopping becomes pocket friendly.
Our route map:







                                                   

Day 1
Day 2
Day 3
Day 4
Day 5
Day 6
Day 7
Day 8
Day 9
Panvel
Trivandrum
Kovalam
Kovalam
Thekkady
Munnar
Munnar
Kochi
Pune

Why should you go by train ?

Konkan railways is different from all others. Amazing sights . Beautiful waterfalls (some of them were huge). Lots of tunnels. Breathtaking rivers and bridges. Glimpses of the Arabian sea every now and then. It's a journey everyone should make at least once in his lifetime. (We took the Sampark Kranti from Panvel)

Our Journey through photographs

Kovalam Beach Side

Lighthouse standing guard at Kovalam Beach
Walking along the beach is such a joy

Sunset at Kovalam Beach



Alleppey Beach beside our resort
Sunset at Alleppy beach

A tiny boat with the sunset in background..Just amazing view
What's Kerala without Coconuts

Kerala offers various kinds of Coconuts. The water and the pulp inside are both consumed
Different kinds of coconuts can be found everywhere.

Kanyakumari

Waiting for sunrise at Kanyakumari was such a unique experience..One side Indian Ocean, another side..The Arabian Sea
Poovar Backwaters

Poovar Backwater was a good experience.

Thekkady/Periyar

An hour long boat ride in the forest. We saw an elephant herd.
Periyar Wildlife Sanctuary
Munnar 

Munnar Tea Gardens

Kochi beach with Chinese fishing nets



The harbor at Kochi is a busy one. You can see huge ships crossing it, all day and night.

The oil industry can be seen at the right corner. Ships keep crossing every now and then at Kochi Harbor

We finally got caught in rain at Kochi . It nevertheless made the scene more captivating.

Huge ship entering , while a couple leisurely walks by.

Very Important Tips



Before you journey starts
  • Book for trains from the list of Konkan Railways that suits best for you. (You can refer this link)
  • Book your train at least 2 months before. There's plenty of rush in these trains.
  • Plan your trip from the bottom of Kerala to the top (Hence,  book train till Trivandrum i.e Ernakulam Railway Station) . Catch return flight from Kochi. 
  • Book the train and flight YOURSELF. DO NOT let a travel company book them for you.
  • For each city you are going to visit, note down good hotels fitting your budget from good websites like TripAdvisor or MakeMyTrip. Then book your vacation from makemytrip or similar travel organizer. Tell them, that you have already bought tickets. They need to plan according to you. Also handover the list of hotels , so they book only from your chosen list.


What to Pack (The most important things)

  • Rain gear (Umbrella, raincoat, lots of plastic bags)
  • Rain proof Foot gear (My Quechua all weather sandals were enough for the whole tour.), Extra bathroom flip flops for beaches.
  • Extra fold-able bag to bring back all shopped items.
  • Laptop (A DSLR is going to need constant flushing of photos)
  • One light jacket (Munnar might be chilly)
  • Towel, Swimming gear, Cap.
  • Binocular.

Locations


Trivandrum:

Visit the Shangumugham Beach same day as you arrive, at evening. Preferably, book the "Uday Suites" hotel (Book online. Use as much vouchers as possible) . From there, it's walking distance to the beach.

Kanyakumari: 

Plan your Kanyakumari trip from Kovalam. Start at night 2 AM , so you can reach before Sun Rise.

Backwater experience:
We suggest, you visit the Poovar Backwater cruise. It's worthy, very private and beautiful. Visit this link for an idea.
Beware: Bargain very well. Take the 1 hr trip ONLY. Any longer is not worth. Pay only max 1000rs per head. No more.

Alleppey:
Choose for a resort along the beach side. Let it be far from the city. You will love the stay. Nothing but clean sea shores, beautiful sun sets. And the sound of the roaring sea. (Please carry mosquito repellents too).

Suggestion: If possible, do not choose to live on a houseboat. I heard it is highly boring and costly. Not worth it. It's your choice. 
Note : You can still visit them for some time, if not stay.

Munnar:
Choose a hotel with a great view. Read review comments before you book. 
Visit the Tata Tea factory. (There's nothing to eat there, so have your breakfast before you start).

Thekkady:
While in the city, choose a hotel in the midst of the market. Stroll around the market and you will get an idea of the richness of the local market in spices, chocolates, and art. The Martial Arts and Dance show is also nearby centrally placed. Shows start at evening . Around 200 rupees per head per show. Don't miss that.

Kochi:
Stay at a hotel near the sea beach. It feels European. It's fun to walk around the coast. Lots of classy hotels to dine. Also check timings for all visiting spots (churches, museum or zoo are closed on some odd days). While going to airport, ask the driver to cross the sea on steamer (the car itself rides on it).

Shopping

This might be the favorite part of many people. Kerala has lots to look out for. 

Saaris and other clothings: From Kochi, or Trivandrum
Chocolates: From Thekkady and Munnar.
Banana chips: Anywhere.
Spices: Thekkady and Munnar
Tea: From munnar local market.
Artifacts: They are very costly. So bargain well.

In all, enjoy the most. Eat, explore, sit beside the shore, and gaze. Bon Voyage.

Kovalam Beach road. Watch the amazing sunset.


Regular Expressions with Python: Examples of common functions

Regular Expressions are tools to filter and grab information from a heap of strings. Python has got a lot of toys to deal with it. The module "re". I am going to give you a quick ride through it. The ones that we use/gonna use the most.

So just immediately launch your favourite IDE and type "import re". Let's get started.

re.search:


for the object r where r = re.search(pattern, subject),

r.start() - Gives the start point where the match was found.
r.end() - Gives the end point where the match was found.
r.group(0) - Returns the full matched string
r.group(1) - Returns the matched string for the 1st group
r.group(2) - Returns the matched string for the 2nd group

And so on....

Note : r.group() is same as r.group(0)

The number of groups available depends on the number of groups used in your pattern.

Example:


>>> regex = r"([a-zA-Z]+) (\d+)"
>>> re.search(regex, 'July 4')
<_sre.SRE_Match object at 0x0000000002E4D140>
>>> r = re.search(regex, 'July 4')
>>> r.end()
6
>>> r.start()
0
>>> 'July 4'[0:6]
'July 4'
>>> 'July 4'[0:5]
'July '
>>> r.group(0)
'July 4'
>>> r.group(1)
'July'
>>> r.group(2)
'4'
>>> r.group(3)

Traceback (most recent call last):
  File "<pyshell#12>", line 1, in <module>
    r.group(3)
IndexError: no such group

re.findall


>>> r = re.findall(regex, 'July 4 August 10 Sept')
>>> r
[('July', '4'), ('August', '10')]

Same example with findall...observe how we got two results since there are two patterns in the string .

Noteworthy : If you are using groups with findall, then the result will be a list containing tuples as above.
But if you are just using single regex (no grouping), the result will be a list with strings .

Example:


>>> re.findall('\w+', 'July 4 August 10')
['July', '4', 'August', '10']

re.finditer

The finditer returns an iterator of re.MatchObjects.


>>> it = re.finditer('\w+', 'July 4 August 10')
>>> for i in it:
        print 'July 4 August 10'[i.start():i.end()]

>>> for i in it:
       print i.group()

 
July
4
August
10

re.compile

If you want to repeat searching with a common pattern , its better to create a separate object (known as compiled regular expression object).

Python official help says:

Compile a regular expression pattern, returning a pattern object.


>>> patt = re.compile('abc')


Lets use the 'patt' object now.


>>> r = patt.search('abcXYZabc1234hjhjkabc')
>>> r.group()
'abc'
>>> r.start()
0
>>> r.end()
3

Things to note:
  • Search only needs a target search string as argument. 
  • Functions start() and end() returns the start index number , and end return the position as i + 1 starting from 0.
Now what if you want to know if there are more matches possible in the same string with our pattern.

>>> r = patt.search('abcXYZabc1234hjhjkabc', 1)
>>> r.group()
'abc'
>>> r.start()
6
>>> r.end()
9

As you can see, I can now traverse in the string to find all possible matches.

re.sub

If you want to replace some text that matches a certain pattern , the re.sub functions comes handy.


So by the syntax, we need to provide a pattern , then a text to be replaced if a match is found, and a target string. Let's see a simple example.


>>> string = "Let have a big party on my birthday"
>>> re.sub('(?<=\s)\w', 'X', string)
"Let's Xave X Xig Xarty Xn Xy Xirthday"

Explanation:

  • We have a sentence. I want to replace all first letters with some letter only if that letter has a space before it. 
  • The pattern : (?<=\s)\w finds all letters that have a space before it. 
  • 'x' is the thing to be replaced with.
  • The variable "string" is the target. 
  • The result shown is the return of the re.sub(). It needs to be stored if you want to use it. The variable "string" remains unchanged.


Total Pageviews