Python's Getters and Setters

You can read a better formatted version of this post here.

Just learn a new python feature called @property. Let's say we create a class called Circle as follows.

class Square:
    side = 2
    area = side * side
s = Square()

Now:

>>> s.side
2
>>> s.area
4

However:

>>> s.side = 3
>>> s.side
3
>>> s.area
4

Not cool! How to solve that?

class Square:
    side = 2
    @property
    def area(self)
        return self.side * self.side
s = Square()

See now.

>>> s.side = 3
>>> s.side
3
>>> s.area
9

Introduction to Data Science

The AskDeveloper group organized this hangout about Data Mining, and Machine Learning in particular. The video is in Arabic, yet the slides are in English.

،شوية دردشة عن تنقيب وتحليل البيانات والتعلم الآلي، الفيديو بالعربي

إضغط على الصورة لمشاهدة تسجيل الحلقة

A Quick Intro. to NumPy

For me, NumPy is a Python list on steroids. You can use it to create multidimensional arrays and matrices.

The convention is to import it as follows:

import numpy as np

To create an array of numbers between 0 and 9, you could use the following command:

x = range(9)

To convert that list into a NumPy array, you can write:

x = np.array(range(9))

And to make you life easier, there is a shorthand for the above command:

x = np.arange(9)

So far, we have been creating one dimensional array. However, there are ways to reshape the arrays. The reshape() method when applied on an array, it returns a reshaped version of it without changing the original object. To reshape the original object itself, then use resize() instead.

y = x.reshape(2,5)

The above command create a 2-dimensional array of 2 rows and 5 columns. You can create a much dimensional arrays as you want. See the command below for a 3*4*5 array.

y = np.arange(3*4*5).reshape(3,4,5)

The mathematical operations '+', '-', '/' and '*' are applied elementwise.

x = np.arange(10)

# To multiply each element of x by 10
y = x + 10

# To multiply each element of x by itself
y = x + x

To do a Matrix Multiplication though:

# Create a 3 * 5 Matrix
A = np.arange(15).reshape(3,5)

# Create a 5 * 2 Matrix
B = np.arange(10).reshape(5,2)

# Dot product gives you a 3 * 2 Matrix
y = y = np.dot(A, B)

Just like lists, you can get parts of arrays

For original lists:

A = range(10)
A[2:5] # [2, 3, 4]

For NumPy Arrays

B =  arange(10)
B[2:5] # array([2, 3, 4])

However, you can set some elements of the array as follows

B[2:5] = 1337

But, you cannot do the same to lists.

A[2:5] = 1337 # TypeError: can only assign an iterable

For statisticians, there are also the following functions

x = np.arange(5) + 1
x.mean() # 3.0
x.max() # 5
x.min() # 1
x.std() # 1.414

You can also access elements of the array using start, stop and a step:

x = np.arange(10)
x[2:7:2] # array([2, 4, 6])

Or access specific elements, let's say elements 1, 5 and 6

x[[1,5,6]] # array([1, 5, 6])

Similar to reshape() and resize(), ravel() converts a multidimensional array into a one-dimensional array, while transpose() turns rows into columns and vice versa.

If you program in R, you will not miss their way of accessing elements of array that meet a certain condition.

x = np.arange(10)
x[x>4] # array([5, 6, 7, 8, 9])
x[x%2 == 1] # array([1, 3, 5, 7, 9])

If you are having an array of elements that are either True or False.

x = np.array([True, False, True, True])

x.all() # Only True if all elements are True
x.any() # Only True if any elements are True

Finally, there is a repeat() that repeats each element of the array n times

x = np.array([1, 2])
x.repeat(3) # array([1, 1, 1, 2, 2, 2])

That's all folks for today.
Check the following tutorial for more information.

MacPorts FTW

Being new to Mac OS X, I am still struggling to memorize its different keyboard shortcuts and to make my fingers tell the difference between the command and control button. However, another thing I have been struggling with lately is, installing my favourite python modules on it. I had some problems installing Scipy and Scikit-learn in particular. I gave Homebrew a try among other things, but it came out that MacPorts was the only way to install those modules successfully.

It's simple and straight forward.

Download and install the MacPorts package for your system here.

On the command line, type the following commands

$ sudo port install python
$ sudo port install py-scipy
etc...

You may aso specify a specific version for python to use, so, type 'python27' and 'py27-scipy' instead. For sure you can use it to download hundreds of other packages, see the list here.

If you, like me, messed a bit with your system paths, make sure to have a look at your '.bash_profile', or whatever shell you use, to see if any other paths set by brew are taking precedence or something.

There is also 'sudo port selfupdate' to make sure all you packages are up-to-date. Check the guide for other useful commands here.

Linkedin's Skills Endorsement is flawed

Linkedin's Skills Endorsement is flawed. I love the idea, but the problem is that the way it suggests for others to endorse their connections is flawed. They only show you the top endorsed skill for one of your friend and invite you to endorse it, and what happens is that (99% of the time), you will just follow their suggestion, rather than selecting the skills your connection really has. In other words, the rich (skills) get richer, and the poor (skills) get poorer.

Another problem, is that I am doing a career shift at the moment, but because most of my connections so far either know me Network Security Engineer or Blogger, my top endorsements are Juniper, Social Media and Networks Security, followed by ones like Firewalls and Twitter. For sure it reflects my expertise, at least the past one, but I would love to see some balance there, and see more endorsements for skills like Python, Machine Learning, Information Retrieval, Data Science, Statistics and the other skills at the tail of my list. One way is form more connection in those new fields so they can reflect my true skills now, but back to the first problem, the Linkedin suggestion system will keep them buried and it will be harder for those skills to get promoted as I want.

Hope they change such suggestion system one day, so my profile reflects what I really am at the moment.

Python Profiler

Was writing a code today and it was so unoptimized and slow. A friend of mine told me about the Profilers. They are a nice way to see which parts of your code are swallowing all of the execution time to work on optimizing them later on.

Here is how to use 'cProfile' in Python. Let's say your main function is called 'main()', so instead of directly calling it, write down the following code:

import cProfile
import pstats
cProfile.run('main()','my_prof')
p = pstats.Stats('my_prof')
p.sort_stats('time').print_stats(10) 

By the way, it came out that 'math.pow(x,2)' was one of the bad guys here, and replacing it with a simple 'x*x' improved the performance a bit.

Python vs Ruby - String to Integer

I'm a big Python fan, but recently, I decided to have a look at Ruby. I'm still not that qualified to do a comparison between the two languages, so let me instead take a snapshot from each of them and do a quick comparison based on it, and based on my own understanding of each.

Both languages have their ways to transform a strings with integers in there into integers.

In Python:

>>> int("33")
33
>>> int("33 + 3")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: invalid literal for int() with base 10: '33 + 3'
>>> int("Three")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: invalid literal for int() with base 10: 'Three'

In Ruby:

irb(main):007:0> "33".to_i
=> 33
irb(main):008:0> "33 + 3".to_i
=> 33
irb(main):009:0> "Three".to_i
=> 0

So first of all, Ruby seems to be more strictly Object Oriented language, than Python. Sure, Python treats everything as an Object too, a String or an Integer are objects in both languages and they have their own properties and methods, which is not the case with C for example, however I see here that Python uses a built-in methods, while Ruby - which also has its built-in methods - yet it tended more to do the transformation here the object way. 

Python prefers to have one - and preferably only one - obvious way to do things, hence when I typed stings composed of stuff other then integers it returned an error. It tends to make it easier for programmers to predict the result. While in Ruby, the result wasn't that obvious, it tried not to return an error and decided to think in an work-around for me me, on the second line it just took the integer part at the beginning of the string then ignored the rest of it, while in the third line it returned zero. At the end of the day, this is a matter of taste, some people might like the Python way of keeping everything predictable, while some others might like the Ruby easy-goingness and not nagging and returning errors to them all the time.

A Waitress & A Flask

Flask is a microframework for Python based on Werkzeug, Jinja2, and here is a simple Flask Hello World code:

from flask import Flask
app = Flask(__name__)

@app.route("/")
def hello():
    return "Hello World!"

if __name__ == "__main__":
    app.run()

Waitress on the other hand is a pure-Python WSGI server, and here's normal usage of the server:

from waitress import serve
serve(wsgiapp, host='0.0.0.0', port=8080)

So, here is how I wrote code to run Flask from within a Waitress WSGI Server:

from flask import Flask
from waitress import serve

app = Flask(__name__)

@app.route("/")
def hello():
 return "Hello World!"

if __name__ == "__main__":
 #app.run()
 serve(app)

Documentations:
http://flask.pocoo.org/docs/
https://github.com/Pylons/waitress/blob/master/docs/index.rst

CherryPy Custom Authentication

You can read a better formatted version of this post here.

While working on Baralbait's API - yes, we may have an API someday, and you can sure contact us if you need to know more about it. Anyway, while developing the API, we were planning to use CherryPy Basic Authentication, in order to authenticate the API Calls.

This is how to add CherryPy Basic Authentication to one of your methods/pages:

import cherrypy

users = {"user1": "secret1", "user2": "secret2"}

def clear_text(mypass):
    return mypass
 
class MyServer:

    def index(self):
        return "This is a public page!"
    index.exposed = True

    def secret(self)
        print "Logged user is:", cherrypy.request.login
        return "Awesome, you've just accessed a page that requires HTTP Basic Authentication"
    secret.exposed = True
    secret._cp_config = {'tools.basic_auth.on': True,
        'tools.basic_auth.realm': 'My Secure Server',
        'tools.basic_auth.users': users,
        'tools.basic_auth.encrypt': clear_text}
 
if __name__ == '__main__':

    cherrypy.quickstart(MyServer(),"/","myserver.conf")

The above code means, that the page called "index", is a public page, while "secret" requires HTTP Basic Authentication, with the credentials mentioned in the "users" dictionary. For more info, Jim Hoskins has an awesome tutorial about using HTTP Basic Authentication in CherryPy, however his site is down now :(

Now, the problem with the above code, is that you can either make a certain page public or secured, but you cannot make it public and private in the same time. Ok, please be patient, let's say that you want authenticated users to see certain content when visiting our secret page, while unauthenticated users should see different content instead of being blocked. For example, we want authenticated users to see their friend's news feed, while unauthenticated users see public news.

So, here comes the beauty of CherryPy Custom tools. You can now, build your own authentication hook, and make it return a null or custom user id when an incorrect or no username or password are given, instead of totally blocking the user.

And here are the modifications needed to the above code:

import cherrypy

from cherrypy.lib import httpauth

users = {"user1": "secret1", "user2": "secret2"}

def clear_text(mypass):
    return mypass

def my_basic_auth(realm, users, encrypt=None):
    if cherrypy.lib.auth.check_auth(users, encrypt):
        print "DEBUG: Authenticated"
        return
    else:
        print "DEBUG: Not Authenticated"
        cherrypy.request.login = "Anonymous"
        return
 
class MyServer:

    def index(self):
        return "This is a public page!"
    index.exposed = True

    def secret(self)
        if cherrypy.request.login == "Anonymous":
            return "This is another public page on our useless website."
        else:
            return "Can you keep a secret, this page is really confidential."
    secret.exposed = True
    secret._cp_config = {'tools.my_basic_auth.on': True,
        'tools.my_basic_auth.realm': 'My Secure Server',
        'tools.my_basic_auth.users': users,
        'tools.my_basic_auth.encrypt': clear_text}
 
if __name__ == '__main__':

    cherrypy.tools.mybasic_auth = cherrypy.Tool('on_start_resource', my_basic_auth)
    cherrypy.quickstart(MyServer(),"/","myserver.conf")

So, we have just created a custom tool, and hooked it in the earliest hook ever, 'on_start_resource', i.e. during the request. We also created our own authentication method, 'my_basic_auth', and attached it to the tool. In our authentication method, which is almost identical to CherryPy's built in HTTP Basic Authentication method, however we do not raise any errors regardless the user is connected or not, we just set 'cherrypy.request.login' to an arbitrary user, that our application can understand later on, such as 'Anonymous'.

Sqlite - Date and Time Data Types

SQLite handles dates quite strangely. If you create a table and put a date in it, the only way SQLite will take the input is "2004-06-01 12:00:00" anything else it won't recognize as a date and thus none of the date functions will work. Assuming you have a date entered propely (like above) you get date information from the db like so.

SELECT date(DateField) FROM Table; SELECT time(DateField) FROM Table; SELECT datetime(DateField) FROM Table;

If you want unix time (seconds since the epoch) you have to format the output.

SELECT strftime("%s",DateField) FROM Table;

However that will return the time in UTC which is probably not what you want (it's not what I wanted). I want it to compensate for my local timezone and thus you have to tell it to use your timezone.

SELECT strftime("%s",DateField,'localtime') FROM Table;

Related Posts:
Green Data: Pysqlite

Sources: Perturb.org , Sqlite Wiki
Tags: Sqlite, Database, Gr33n Data

PyFusker

Python implementation of Fusker Introduction: Most probably you are redirected from Wikipedia to this page, so please if you have any comment on the code just leave it here so that I can improve it. Also if you can help me know why are people interested in the fusker technology that much, and what web sites do you usually fusker.

The Python Code
#!/usr/bin/python
""" PyFusker """
""" Author: Tarek Amr """
""" Version: 0.1 """
""" File: pyfusker.py """
""" Date: 07 January 2006 """
""" Replce [TAB] with \t """
class Fusker:
[TAB]def __init__(self, url):
[TAB][TAB]self.url = url
[TAB][TAB]self.urls = []
[TAB][TAB]self._generate()
[TAB]def _generate(self):
[TAB][TAB]comp = re.compile("\\[.+\\]")
[TAB][TAB]x = comp.findall(self.url)
[TAB][TAB]if not x:
[TAB][TAB][TAB]print "Url parsing error!"
[TAB][TAB][TAB]sys.exit()
[TAB][TAB]try:
[TAB][TAB][TAB]x[0] = x[0].strip("[")
[TAB][TAB][TAB]x[0] = x[0].strip("]")
[TAB][TAB][TAB]x1,x2 = x[0].split("-")
[TAB][TAB][TAB]x1 = string.atoi(x1)
[TAB][TAB][TAB]x2 = string.atoi(x2)
[TAB][TAB]except:
[TAB][TAB][TAB]print "Url parsing error!"
[TAB][TAB][TAB]sys.exit()
[TAB][TAB]if x1 > x2:
[TAB][TAB][TAB]print "Invalid range"
[TAB][TAB][TAB]sys.exit()
[TAB][TAB]temp_url = comp.split(url)
[TAB][TAB]i = x1
[TAB][TAB]while i < x2 :
[TAB][TAB][TAB]self.urls.append(string.join(temp_url,"%s" % i))
[TAB][TAB][TAB]i = i + 1
[TAB][TAB]return self.urls
[TAB]def list_urls(self):
[TAB][TAB]print self.urls
[TAB]def download_urls(self):
[TAB][TAB]for u in self.urls:
[TAB][TAB][TAB]try:
[TAB][TAB][TAB][TAB]print "Downloading ... ", u
[TAB][TAB][TAB][TAB]os.system("wget %s" % u)
[TAB][TAB][TAB]except:
[TAB][TAB][TAB][TAB]print "Wget Error!"
[TAB][TAB][TAB][TAB]continue
if __name__ == "__main__":
[TAB]try:
[TAB][TAB]url = sys.argv[1]
[TAB]except:
[TAB][TAB]print "Usage: python fusker.py http://www.website.com/img[01-06].jpg"
[TAB][TAB]sys.exit()
[TAB]f = Fusker(url)
[TAB]f.download_urls()

Tags: Python, Fusker, Gr33n Data

Pyblish

It is hard to write programs source code on your own blog or webpage especially when writing a language like Python where tabs and white spaces do matter. However blogs are the best place to share your coding ideas and receive comments and bugs fixes on them. So one nice idea is to replace the tab or "\t" with [TAB], and since it is really hard to do it by hand this is a python proggie to replace them automatically.

''' Pyblish - Python Publisher '''
''' Author: Tarek Amr '''
''' Version: 0.1'''
''' File: pyblish.py'''
''' Date: 07 Jan 2006 '''
import string, sys
if __name__ == "__main__":
[TAB]try:
[TAB][TAB]filein = sys.argv[2]
[TAB][TAB]mode = sys.argv[1]
[TAB]except:
[TAB][TAB]print "Usage: python pyblish [ --decode | --encode ] INPUT_FILE_NAME"
[TAB][TAB]sys.exit()
[TAB]if mode == "--decode":
[TAB][TAB]temp = filein.split(".")
[TAB][TAB]temp[-1] = ".txt"
[TAB][TAB]fileout = string.join(temp)
[TAB]elif mode == "--encode":[TAB]
[TAB][TAB]temp = filein.split(".")
[TAB][TAB]temp[-1] = ".py"
[TAB][TAB]fileout = string.join(temp)
[TAB]else:
[TAB][TAB]print "Usage: python pyblish [ --decode | --encode ] INPUT_FILE_NAME"
[TAB][TAB]sys.exit()
[TAB]fdin = open(filein,"r")
[TAB]fdout = open(fileout,"w")
[TAB]for line in fdin.readlines():
[TAB][TAB]if mode == "--decode":
[TAB][TAB][TAB]line = line.replace("\t","[TAB]")
[TAB][TAB]elif mode == "--encode":
[TAB][TAB][TAB]line = line.replace("[TAB]","\t")
[TAB][TAB]fdout.write(line)
[TAB]fdin.close()
[TAB]fdout.close()

By the way this code has nothing to do with that web server and application framework hosted at sourceforge and called Pyblish too

Tags: Python, Blog, Gr33n Data

GMailRCS - Version 1.00

GmailRCS is a way to use your Gmail account as a lightweight Revision Control System (RCS) in order to backup and restore the different versions of your source code. The software is written in Python, using libgmail
GmailRCS: http://sourceforge.net/projects/gmailrcs/


Tags: Gmail, Programming, Gr33n Data

Python - libgmail Tutorial

This is a short tutorla about libgmail. Which is a Library to provide access to Gmail via Python. Tabs are important in python as they indicate the different code blocks, since they do not have "{" and "}" like those in C and C++. So I'll replace the tabs here with [TAB], since it's hard to write the proper format in HTML.

First of all lets see how can we get the libgmail help, a.k.a man pages:

>>>import libgmail
>>>help(libgmail)

Then lets see how can we retrieve the messages in our the inbox:

>>>import libgmail
>>>ga = libgmail.GmailAccount("username@gmail.com", "password")
>>>ga.login()
>>>folder = ga.getMessagesByFolder("inbox", True)
>>>for thread in folder:
>>>[TAB]print "Thread:", thread.id, "Subject:", thread.subject
>>>[TAB]for msg in thread:
>>>[TAB][TAB]print "Msg:", msg.id, ",Author", msg.author, ",Subject:", msg.subject

Now lets say we want to print all the attached files in all the messages labled by "projectx"

>>>import libgmail
>>>ga = libgmail.GmailAccount("username@gmail.com", "password")
>>>ga.login()
>>>folder = ga.getMessagesByLabel("projectx", True)
>>>for tread in folder:
>>>[TAB]for msg in thread:
>>>[TAB][TAB]for attach in msg.attachments:
>>>[TAB][TAB][TAB]print attach._getContent()

You may also send a message this way:

>>>import libgmail
>>>ga = libgmail.GmailAccount("username@gmail.com", "password")
>>>ga.login()
>>>msg=libgmail.GmailComposedMessage("friend@gmail.com", "SubjectHere", "BodyHere")
>>>ga.sendMessage(msg)

To send attachmented files into the message:

>>>import libgmail
>>>ga = libgmail.GmailAccount("username@gmail.com", "password")
>>>ga.login()
>>>myFiles = ["/file1", "/file2", "/file3"]
>>>msg=libgmail.GmailComposedMessage("friend@gmail.com", "SubjectHere", "BodyHere", filenames=myFiles)
>>>ga.sendMessage(msg)

Finally I've noticed that sometimes the sendMessage() function fails - returns "HTTP Error 404: Not Found", and hope that they are going fix it soon. Actually the function fails on the first time but when you call it again it works, so you may use a "try-except" statement like the one below:

>>>try:
>>>[TAB]ga.sendMessage(msg)
>>>except:
>>>[TAB]ga.sendMessage(msg)

Tags: Python, Gmail, Gr33n Data

Del.icio.us Python Popular Links

Want to stay on the cutting edge of what’s hot in the Python community? This is the way to do it. The site tracks the most popular URLs tagged with Python, so you get a continuous feed of interesting and helpful links. I’ve added the RSS feed for the page as a Live Bookmark in Firefox, making it easy to skim the list. If you don’t already have a del.icio.us account, go get one now (although you don’t need one just to view other people’s links).

Source: Import This @ Think Hole Labs

Tags: Programming, Python, Del.icio.us, Gr33n Data

Python Mobile Bluetooth Console

So, now you wanna write Python scripts on your phone, but you are sick of the phone keyboard. Then it is time for BTConsole (Bluetooth Console), where you can have a console connection to the Mobile Phone from your PC using the Bluetooth connection as a Serial Port Emulator. This can be done on both Windows or Linux, but I am going to describe the Linux part as I don't like Windows and also I failed to set it up on it. First of all you have to have the Bluetooth Protocol Stack (BlueZ) installed on your machine. You may also use an "ipconfig" or "ifconfig" like command on you Computer to get information about your Computer's Bluetooth Address (It looks like the MAC Address) and so ... this command is called "hciconfig" Also to discover the Bluetooth devices near you, use "hcitool scan" Now create a Virtual Port (/dev/rfcomm0) on your Linux Machine for the Bluetoot connection and start listening on that port ... Type the following two commands on you computer:

"sdptool add-channel=2 SP" (It shall tell you someting like "Serial Port service registered" now) "rfcomm listen /dev/rfcomm0 2" (It shall tell you something like "Waiting for connection on Channel2")

Then on you Mobile Phone, go to Python (You have Python for Series60 installed on your phone, don't you !?). And choose Bluetooth Console from the Options menu. It should connect to the computer now. And the following will be shown on you computer "Connection from AA:BB:CC:DD:EE:FF to /dev/rfcomm0" then "Press CTRL-C for hangup", where AA:BB:CC:DD:EE:FF is your Mobile's Bluetoot Address The Linux equivalent to HyperTerminal is called minicom, so it's time to use it to open a console connection to the mobile phone. To run minicom type "minicom -s -m" then choose "Serial Port Setup" and set the Serial Device to "/dev/rfcomm0" and leave the rest as it is. Now press escape and exit the Serial Port Setup Menu. Here we go, you are now connected to the Python Shell on your Mobile Phone. If you'd like you may use this to send an SMS from your PC:

>>> import messaging
>>> messaging.sms_send("+20101234567",u"Hello World")

Where +20101234567 is the phone number you want to send your SMS to. Resources:

• Hack a Mobile Phone with Linux and Python
• Nokia Discussion Forum
• Sending SMS from Nokia Series60

By the way if this doesn't work, try the following howto by Dankoozy: HOWTO: Use the Bluetooth console in PyS60

Tags: Nokia, Mobile, Linux, Bluetooth, Python, Gr33n Data