Tag Archives: Python

Python code.

Easy Python JSON Client & Server

The jsocket package is for use during the development of distributed systems. There are two ways to use the package. The first and simplest way is to create a custom single threaded server by overloading the the jsocket.ThreadedServer class. The second, is to use the server factory functionality by overloading the jsocket.ServerFactoryThread class and passing this declaration to the jsocket.ServerFactory(FactoryThread) object. This creates a multithreaded custom JSON server for any number of simultaneous clients.

All of the code can be forked on GitHub here, and the latest release can be installed off PyPi using either “pip install jsocket” or “easy_install jsocket” depending on which python package manager you use.

Using some UML notation you can decipher the inheritance of the jsocket package to be six new module classes and two existing Python classes:

Regardless of which way you choose to use the package, class inheritance is mandatory. As stated above, the simplest way to leverage functionality is to create a custom single threaded server by overloading the ThreadedServer class (just like the ServerFactory class). This is a great way to prototype simple applications in which you want to transfer data in JSON format between a client.

     import jsocket
     import logging
     class MyServer(jsocket.ThreadedServer):
          # This is a basic example of a custom ThreadedServer.
          def __init__(self):
               super(MyServer, self).__init__()
               self.timeout = 2.0

          def _process_message(self, obj):
               # pure virtual method from base class
               if obj != '':
                    if obj['message'] == "new connection":
                         logging.info("new connection")

     if __name__ == "__main__":
          client = jsocket.JsonClient()
          client.connect()
          client.send_obj({"message": "new connection"})

Albeit brief, the above example creates a server that is capable of acting on JSON data of type “{‘message’: ‘new connection’}”. There are no restrictions on what you declare in the _process_message virtual method. You can think of this as defining your ‘protocol’ for the server.

The second approach is to use the ServerFactory class to create a multithreaded custom JSON server for any number of simultaneous clients. To do this you must inherit from the ServerFactoryThread class. Just like above you need to implement the virtual _process_message method to support your ‘protocol’ (that is, what kind of data you want the server to respond to). After this is done, simply pass the class declaration to the ServerFactory constructor and it does the rest. Whenever the server sees a new client connection it will fork a separate thread of control to manage the interaction with that client (the thread is automatically terminated when the client disconnects).

     import jsocket
     import logging
     class MyFactoryThread(jsocket.ServerFactoryThread):
          # This is an example factory thread, which the server factory will
          # instantiate for each new connection.
          def __init__(self):
               super(MyFactoryThread, self).__init__()
               self.timeout = 2.0

          def _process_message(self, obj):
               # virtual method - Implementer must define protocol
               if obj != '':
                    if obj['message'] == "new connection":
                         logging.info("new connection")
                    else:
                         logging.info(obj)

     if __name__ == "__main__":
          server = jsocket.ServerFactory(MyFactoryThread)
          server.timeout = 2.0
          server.start()

          #create and connect as many clients as you like here
          client = jsocket.JsonClient()
          client.connect()
          client.send_obj({"message": "new connection"})

          client.close()
          server.stop()
          server.join()

Python – Recursive Glob & Line Counter

The other day I needed a recursive glob to find all the *.py files in my home directory. Much to my amazement Python doesn’t have one built into the glob module. So, I built my own and decided to share it for anyone else who might need it. Fork it here and install it with easy_install off PyPi.

"""
Recursive Glob Module
Methods:
	rglob(base, pattern)
	rglob_(pattern)
	lcount(base, pattern, func=lambda x : True)
"""
import glob
import os

def _getDirs(base):
	return [x for x in glob.iglob(os.path.join( base, '*')) if os.path.isdir(x) ]

def _count(files, func):
	lines = 0
	for f in files:
		lines += sum([1 for l in open(f) if func(l)])
	return lines

def rglob(base, pattern):
	""" Recursive glob starting in specified directory """
	flist = []
	flist.extend(glob.glob(os.path.join(base,pattern)))
	dirs = _getDirs(base)
	if len(dirs):
		for d in dirs:
			flist.extend(rglob(os.path.join(base,d), pattern))
	return flist

def rglob_(pattern):
	""" Performs a recursive glob in the current working directory """
	return rglob(os.getcwd(), pattern)

def lcount(base, pattern, func = lambda x : True):
	""" Counts the number of lines in each file found matching pattern.
		Params:
			base - root directory to start the search
			pattern - pattern for glob to match (i.e '*.py')
			func - boolean filter function
				example: lambda x : True if len(x.strip()) else False #filter empty lines
				default: lambda x : True
	"""
	allFiles = rglob(base, pattern)
	return _count(allFiles, func)

if __name__ == "__main__":
	#filter out empty lines and comments
	filterFunc = lambda x : True if (len(x.strip()) and x.strip()[0] != '#') else False

	pyFiles = rglob(os.path.dirname(__file__), "*.py")
	print " {} total lines".format(_count(pyFiles, filterFunc))

	pyFiles_ = rglob_("*.py")
	print " {} total lines".format(_count(pyFiles_, filterFunc))
	print " {} total lines".format(lcount(os.path.dirname(__file__), "*.py", filterFunc))

It works by starting off in a base directory where it uses a generator expression to iterate through a glob iterator that matches any subdirectory. It then goes through these subdirectories one at a time and performs a glob pattern match on the contents of said directory. This is repeated recursively until all subdirectories off the base are inspected.

Python Generators

Generators are an easy-to-use tool in Python for making any object method or class an iterator. I’ve found them to be a very useful development tool and wanted to briefly expand on my previous PyQt Charts post.

The simplest generator is an object method that uses the “yield” keyword. Yield is just like return only the instruction counter is persistent in the method. In the example below, a generator method is created to return characters of a string one at a time in reverse order. A more real world example would be making a read socket a generator that returns packet data.

def reversed(s):
    index = len(s)
    while index >= 0:
        yield s[index]
        index = index - 1

revGen = reversed("Android")
for c in revGen:
     print c

The second and more useful approach is to let classes be generators. This is my preferred method because OOD is always better :-). Every class definition can overload the __iter__ and next methods. When using the class approach the StopIteration exception is responsible for terminating the iteration.

class QCharts(object):
	def __init__(self, **kwargs):
		self.charts = []
		self.index = 0
		self.len = 0

	def __iter__(self):
		self.index = 0
		return self

	def next(self):
		if self.index == self.len:
			raise StopIteration
		self.index = self.index + 1
		return self.charts[self.index-1]

	def add(self, c):
		self.charts.append(c)
		self.len = len(self.charts)

Now, using this iterator class we can loop through a series of QWidget based charts. Below is a slightly modified MainWindow class from the PyQt Charts post.


from PyQt4 import QtCore, QtGui
from pygooglechart import PieChart3D

class QChart(parent, type, **kwargs):...
class QCharts(object):...	

class MainWindow(QtGui.QMainWindow):
	def __init__(self, **kwargs):
		super(QtGui.QMainWindow, self).__init__(**kwargs)
		self.charts = QCharts()

	def addPieChart(self):
		i = self.charts.len
		t = QChart(self, PieChart3D, pos=QtCore.QPoint(i*75,i*75), size=QtCore.QSize(250,100))
		t.add_data([(20*i)+10,(i*10)+20])
		t.set_pie_labels(['Hello', 'World'])
		t.download()
		self.charts.add(t)

	def debug(self):
		for x in self.charts:
			print x

if __name__ == '__main__':
	app = QtGui.QApplication(sys.argv)
	app.setApplicationName("Blank PyQt Demo")
	app.setQuitOnLastWindowClosed(True)

	scaledSize = QtCore.QSize(scale.width(500),scale.height(500))
	window = MainWindow(size=scaledSize)
	window.addPieChart()
	window.addPieChart()
	window.addPieChart()
	window.debug()
	window.setWindowTitle("Immersive Blank PyQt Demo")
	window.show()

	sys.exit(app.exec_())

For more information on generators check out this great write up here. Another great feature of generators is that they can be used in generator expressions. These expressions look like this:

#extract the even numbers from the series
ints = [0,1,2,3,4,5,6,7,8,9,10]
even = [x for x in ints if x % 2 == 0 ]

#get the directory names from glob
import glob
def dirs(p):
     return [x for x in glob.iglob(os.path.join( p, '*')) if os.path.isdir(x) ]
dirs("/")

JavaScript & Python Closures

Closures are generally considered to be JavaScripts most advanced and useful feature, and hence. As a C++ coder I like to think of Closures as a method with a privately malloc’d stack that is persistent across multiple invocations. As a Python coder I like to think of Closures as a method object with a preserved variable environment. Regardless of how you want to think about them, Closures allow us to redefine how we approach coding up methods.

Take the following simple JS example. Here, we are creating a very simple incrementer. The method inc() takes a starting value and returns a method that simply adds one to the start value (but not until its actually executed in the loop below).

function inc(start){
	var i = start;
	return function() { i = i + 1; document.write(i); }
}

addOne = inc(0)
for (i = 0; i < 10; ++i){
	addOne();
}
// prints 12345678910

I find this feature of the language very cool. We can also do the same thing with Python.

#Works with Python 3.x only
def inc(start):
    y = start
    def adder():
        nonlocal y
        y += 1
        print y
        return y
    return adder
 
addOne = inc(0)
x = 0
for x in range(10):
    addOne()
#prints 12345678910

The ramifications of Closures to popular programming languages are still in its infancy even though the concept has been around for decades (adding them to Java has been a debate for years). It will be extremely interesting to see how the wide spread adoption of Closures will ultimately impact our coding habits in the near future.

Will Python ever be taken seriously?

I’ve been writing Python for almost a year now and have been pleasantly surprised by the language. Even though Python has been around for a few decades now it still hasn’t been widely accepted as a production grade programming language.

Its primary use is still in the online world where it has developed a large rivalry with Ruby. It’s been my observation that Ruby wins this match up in the eyes of developers about 80% of the time. In the embedded world the use of Python is almost unheard of and I’ve personally never seen it used exclusively for a full stand alone application.

The language is aesthetically pleasing, requires less typing, and is fantastic for rapid prototyping. It definitely has its quarks and drawbacks but what language doesn’t? When I mention to other developers that I’m currently writing in Python the most common reaction is disbelief. The language just isn’t taken seriously. Why?

PyQt Charts

The Qt framework doesn’t contain any useful charting widgets. There are a few python charting modules out there, most notably PyQwt. As a PyQt developer I didn’t find this module very useful. What I really wanted was a charting widget that used QWidget as its base class. At the time of this post I couldn’t find anything so I wrote my own based on the google charts module.

What I love about this snippet is that it mimics the C++ template functionality in Python. The QChart function below abstracts the PyQtChart class definition and any pygooglechart object can be instantiated as a QWidget.

__author__	  = "Christopher Piekarski"
__email__	  = "@c_piekarski"

import time, os, sys
from PyQt4 import QtCore, QtGui
from pygooglechart import *

def QChart(parent, type, **kwargs):
	class PyQtChart(type, QtGui.QWidget):
		def __init__(self, parent, **kwargs):
			QtGui.QWidget.__init__(self, parent, **kwargs)
			type.__init__(self, kwargs["size"].width(), kwargs["size"].height())
			self.pix = QtGui.QPixmap()

		def download(self):
			file = "./%f.png" % time.time()
			type.download(self, file)
			self.pix.load(file)

		def paintEvent(self, event):
			p = QtGui.QPainter(self)
			p.drawPixmap(0,0,self.pix)
			super(PyQtChart, self).paintEvent(event)

	return PyQtChart(parent, **kwargs)

class MainWindow(QtGui.QMainWindow):
	def __init__(self, **kwargs):
		super(QtGui.QMainWindow, self).__init__(**kwargs)

		t = QChart(self, PieChart3D, size=QtCore.QSize(250,100))
		t.add_data([10,20])
		t.set_pie_labels(['Hello', 'World'])
		t.download()

if __name__ == '__main__':
	app = QtGui.QApplication(sys.argv)
	app.setApplicationName("PyQt Charting Demo")
	app.setQuitOnLastWindowClosed(True)

	scaledSize = QtCore.QSize(500,500)
	window = MainWindow(size=scaledSize)
	window.setWindowTitle("PyQt Charting Demo")
	window.show()

	sys.exit(app.exec_())

Getting the System IP Address with Python

The default Python 2.7.1 distribution has very limited functionality for getting the system IP. If your not willing to install an external module then you really only have one choice… open a socket, establish a connection, and get the active IP from the socket name tuple.

import socket
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.connect(('immersivelabs.com', 0))
s.getsockname()[0]

Although this solution works, it has a few issues. The biggest draw back is that it only returns one of the systems IP addresses. Fortunately, there is a great module called pynetinfo. Install the module using easy_install and then wrap its functionality with the following methods.

import netinfo

def getSystemIfs():
    return netinfo.list_active_devs()

def getSystemIps():
    """ will not return the localhost one """
    IPs = []
    for interface in getSystemIfs():
        if not interface.startswith('lo'):
            ip = netinfo.get_ip(interface)
            IPs.append(ip)
    return IPs

def getIPString():
    """ return comma delimited string of all the system IPs"""
    return ",".join(getSystemIps())

You can now easily get a list or a string containing all the active IP addresses on the system (POSIX Only).

Thrift RPC Framework

At TechStars, Alessio Signorini and I had the opportunity to experiment with Facebook’s Thrift remote procedure call (RPC) framework. I was thoroughly impressed with the framework and feel it takes distributed computing to whole new level. Every developer regularly working on both large and small client/server architectures must be familiar with this tool.

Developers can easily specify data types (parameters) and server interface definitions (API) in a plain text file. Thrift uses this text file to create programming language independent client and server stub files. What makes this so powerful, is that system architects can now choose the appropriate programing language for each area of the system.

Not all programing languages are created equal and no one language contains just positive attributes. It is often the case that the ideal language for a server is Java or C++, but the client would best be written in Python. Thrift allows you to easily choose the best language for the job without having to reinvent the communication protocol every-time you change the language.

This framework is a game changer for software developers and system architects. If you haven’t already done so I strongly encourage you to start playing with it. Below is an example thrift file that defines several data types and one service. Once this interface is defined it can easily be converted into stubs for any language which keeps the interface API separate from the implementation. Read the whitepaper here.

#!/usr/local/bin/thrift --gen cpp:pure_enums --gen php

namespace cpp displaymanager

enum pushResult
{
  OK,
  BUSY,
  INVALID
}

struct pushUrl
{
  1:  string name,
  2:  string url
}

struct pushHtml
{
  1:   string name,
  2:   string data
}

service displaymanager
{
  pushResult fetchUrl(1: pushUrl url);
  pushResult postHtml(1: pushHtml html);
  string	 whatsLoaded();
  string	 currentUrl();
}

Python – Love / Hate

Hate:

1) Really bad at managing threads.
2) All class properties require “self” keyword.
3) Inheritance is supported but can be difficult to invoke.
4) Private members are difficult to encapsulate.
5) More difficult to deploy then traditional compiled languages.
6) Default timer class is limiting.

Love:

1) Tuples.
2) Keyword parameters.
3) Everything is an object.
4) The “pass” keyword.
5) Generator objects.
6) Method objects.
7) Dynamic code insertion.
8) PEP8.

Python – No Enumerated Data Type?

In the embedded world where C/C++ are the predominantly used languages, the enum data type is heavily used. It’s quick and easy to use and understand for specifying flags and register values.  Not yet understanding that Python’s real strength is hash-able data types and that a switch statement didn’t exist, I found myself wanting to use traditional C/C++ enumerated data types when I first started learning Python.

Even though there is no “enum” keyword in Python, there are a few easy ways to create an enumerated data type. The following snippet uses the Python argument packer to create a new data type that is just a key value paired dictionary.

def enum(*args):
    enums = dict(zip(args, range(len(args))))
    return type('Enum', (), enums)

Thats it. This simple function takes a list of items and auto-completes the value. It can easily be used like the following.

alignment = enum("left","center","right")

def alignText(text, pos):
    if pos == alignment.left:
        pass

alignText("some text", alignment.left)

Another object oriented approach I came across is to overload the existing Python set( ) data type and override the __getattr__() method.

class Enum(set):
    def __getattr__(self, name):
        if name in self:
            return name
        raise AttributeError

    def __setattr__(self, name, value):
        pass

    def __delattr__(self, name):
        pass

Neither approach fully addresses all the functionality of the C/C++ enum type. The missing functionality is the ability to specify a value for an entry and having all sequential entries be incremented by one (below). However, both solutions do offer a quick enumerated data type that can be used in most scenarios.

enum alignment = {	LEFT,		// 0 
					CENTER=10,	// 10
					RIGHT		// 11 
		 		 };