Tag Archives: Python

Python code.

Python JSON Client & Server Redux

The jsocket package is for use during the development of distributed systems. There are two ways to use the package. The first and simplest way is to create a custom single threaded server by overloading the the jsocket.ThreadedServer class. The second, is to use the server factory functionality by overloading the jsocket.ServerFactoryThread class and passing this declaration to the jsocket.ServerFactory(FactoryThread) object. This creates a multithreaded custom JSON server for any number of simultaneous clients.

All of the code can be forked on GitHub here, and the latest release can be installed off PyPi using either “pip install jsocket” or “easy_install jsocket” depending on which python package manager you use.

Using some UML notation you can decipher the inheritance of the jsocket package to be six new module classes and two existing Python classes:

Regardless of which way you choose to use the package, class inheritance is mandatory. As stated above, the simplest way to leverage functionality is to create a custom single threaded server by overloading the ThreadedServer class (just like the ServerFactory class). This is a great way to prototype simple applications in which you want to transfer data in JSON format between a client.

     import jsocket
     import logging
     class MyServer(jsocket.ThreadedServer):
          # This is a basic example of a custom ThreadedServer.
          def __init__(self):
               super(MyServer, self).__init__()
               self.timeout = 2.0

          def _process_message(self, obj):
               # pure virtual method from base class
               if obj != '':
                    if obj['message'] == "new connection":
                         logging.info("new connection.")

     if __name__ == "__main__":
          client = jsocket.JsonClient()
          client.connect()
          client.send_obj({"message": "new connection"})

Albeit brief, the above example creates a server that is capable of acting on JSON data of type “{‘message’: ‘new connection’}”. There are no restrictions on what you declare in the _process_message virtual method. You can think of this as defining your ‘protocol’ for the server.

The second approach is to use the ServerFactory class to create a multithreaded custom JSON server for any number of simultaneous clients. To do this you must inherit from the ServerFactoryThread class. Just like above you need to implement the virtual _process_message method to support your ‘protocol’ (that is, what kind of data you want the server to respond to). After this is done, simply pass the class declaration to the ServerFactory constructor and it does the rest. Whenever the server sees a new client connection it will fork a separate thread of control to manage the interaction with that client (the thread is automatically terminated when the client disconnects).

     import jsocket
     import logging
     class MyFactoryThread(jsocket.ServerFactoryThread):
          # This is an example factory thread, which the server factory will
          # instantiate for each new connection.
          def __init__(self):
               super(MyFactoryThread, self).__init__()
               self.timeout = 2.0

          def _process_message(self, obj):
               # virtual method - Implementer must define protocol
               if obj != '':
                    if obj['message'] == "new connection":
                         logging.info("new connection.")
                    else:
                         logging.info(obj)

     if __name__ == "__main__":
          server = jsocket.ServerFactory(MyFactoryThread)
          server.timeout = 2.0
          server.start()

          #create and connect as many clients as you like here
          client = jsocket.JsonClient()
          client.connect()
          client.send_obj({"message": "new connection"})

          client.close()
          server.stop()
          server.join()

Python – Recursive Glob & Line Counter

The other day I needed a recursive glob to find all the *.py files in my home directory. Much to my amazement Python doesn’t have one built into the glob module. So, I built my own and decided to share it for anyone else who might need it. Fork it here and install it with easy_install off PyPi.

"""
Recursive Glob Module
Methods:
	rglob(base, pattern)
	rglob_(pattern)
	lcount(base, pattern, func=lambda x : True)
"""
import glob
import os

def _getDirs(base):
	return [x for x in glob.iglob(os.path.join( base, '*')) if os.path.isdir(x) ]

def _count(files, func):
	lines = 0
	for f in files:
		lines += sum([1 for l in open(f) if func(l)])
	return lines

def rglob(base, pattern):
	""" Recursive glob starting in specified directory """
	flist = []
	flist.extend(glob.glob(os.path.join(base,pattern)))
	dirs = _getDirs(base)
	if len(dirs):
		for d in dirs:
			flist.extend(rglob(os.path.join(base,d), pattern))
	return flist

def rglob_(pattern):
	""" Performs a recursive glob in the current working directory """
	return rglob(os.getcwd(), pattern)

def lcount(base, pattern, func = lambda x : True):
	""" Counts the number of lines in each file found matching pattern.
		Params:
			base - root directory to start the search
			pattern - pattern for glob to match (i.e '*.py')
			func - boolean filter function
				example: lambda x : True if len(x.strip()) else False #filter empty lines
				default: lambda x : True
	"""
	allFiles = rglob(base, pattern)
	return _count(allFiles, func)

if __name__ == "__main__":
	#filter out empty lines and comments
	filterFunc = lambda x : True if (len(x.strip()) and x.strip()[0] != '#') else False

	pyFiles = rglob(os.path.dirname(__file__), "*.py")
	print " {} total lines".format(_count(pyFiles, filterFunc))

	pyFiles_ = rglob_("*.py")
	print " {} total lines".format(_count(pyFiles_, filterFunc))
	print " {} total lines".format(lcount(os.path.dirname(__file__), "*.py", filterFunc))

It works by starting off in a base directory where it uses a generator expression to iterate through a glob iterator that matches any subdirectory. It then goes through these subdirectories one at a time and performs a glob pattern match on the contents of said directory. This is repeated recursively until all subdirectories off the base are inspected.

Python Generators

Generators are an easy-to-use tool in Python for making any object method or class an iterator. I’ve found them to be a very useful development tool and wanted to briefly expand on my previous PyQt Charts post.

The simplest generator is an object method that uses the “yield” keyword. Yield is just like return only the instruction counter is persistent in the method. In the example below, a generator method is created to return characters of a string one at a time in reverse order. A more real world example would be making a read socket a generator that returns packet data.

def reversed(s):
    index = len(s)
    while index >= 0:
        yield s[index]
        index = index - 1

revGen = reversed("Android")
for c in revGen:
     print c

The second and more useful approach is to let classes be generators. This is my preferred method because OOD is always better :-). Every class definition can overload the __iter__ and next methods. When using the class approach the StopIteration exception is responsible for terminating the iteration.

class QCharts(object):
	def __init__(self, **kwargs):
		self.charts = []
		self.index = 0
		self.len = 0

	def __iter__(self):
		self.index = 0
		return self

	def next(self):
		if self.index == self.len:
			raise StopIteration
		self.index = self.index + 1
		return self.charts[self.index-1]

	def add(self, c):
		self.charts.append(c)
		self.len = len(self.charts)

Now, using this iterator class we can loop through a series of QWidget based charts. Below is a slightly modified MainWindow class from the PyQt Charts post.


from PyQt4 import QtCore, QtGui
from pygooglechart import PieChart3D

class QChart(parent, type, **kwargs):...
class QCharts(object):...	

class MainWindow(QtGui.QMainWindow):
	def __init__(self, **kwargs):
		super(QtGui.QMainWindow, self).__init__(**kwargs)
		self.charts = QCharts()

	def addPieChart(self):
		i = self.charts.len
		t = QChart(self, PieChart3D, pos=QtCore.QPoint(i*75,i*75), size=QtCore.QSize(250,100))
		t.add_data([(20*i)+10,(i*10)+20])
		t.set_pie_labels(['Hello', 'World'])
		t.download()
		self.charts.add(t)

	def debug(self):
		for x in self.charts:
			print x

if __name__ == '__main__':
	app = QtGui.QApplication(sys.argv)
	app.setApplicationName("Blank PyQt Demo")
	app.setQuitOnLastWindowClosed(True)

	scaledSize = QtCore.QSize(scale.width(500),scale.height(500))
	window = MainWindow(size=scaledSize)
	window.addPieChart()
	window.addPieChart()
	window.addPieChart()
	window.debug()
	window.setWindowTitle("Immersive Blank PyQt Demo")
	window.show()

	sys.exit(app.exec_())

For more information on generators check out this great write up here. Another great feature of generators is that they can be used in generator expressions. These expressions look like this:

#extract the even numbers from the series
ints = [0,1,2,3,4,5,6,7,8,9,10]
even = [x for x in ints if x % 2 == 0 ]

#get the directory names from glob
import glob
def dirs(p):
     return [x for x in glob.iglob(os.path.join( p, '*')) if os.path.isdir(x) ]
dirs("/")

JavaScript & Python Closures

After a hard day of climbing in the canyon my fellow climbing/coder friend Tim started to tell me about a feature of JavaScript he had just started to use called Closures. Even though I have written thousands of lines of JavaScript I had never heard of Closures. As a self proclaimed eternal student I was intrigued with what he was telling me.

Turns out that Closures are generally considered to be JavaScripts most advanced and useful feature, and hence, is worthy of a blog post :-). As a C++ coder I like to think of Closures as a method with a privately malloc’d stack that is persistent across multiple invocations. As a Python coder I like to think of Closures as a method object with a preserved variable environment. Regardless of how you want to think about them, Closures allow us to redefine how we approach coding up methods.

Take the following simple JS example. Here, we are creating a very simple incrementer. The method inc() takes a starting value and returns a method that simply adds one to the start value (but not until its actually executed in the loop below).

function inc(start){
	var i = start;
	return function() { i = i + 1; document.write(i); }
}

addOne = inc(0)
for (i = 0; i < 10; ++i){
	addOne();
}
// prints 12345678910

I’m probably just super geeky but I find this feature of the language very cool. Turns out we can also do the same thing with Python.

#Works with Python 3.x only
def inc(start):
    y = start
    def adder():
        nonlocal y
        y += 1
        print y
        return y
    return adder
 
addOne = inc(0)
x = 0
for x in range(10):
    addOne()
#prints 12345678910

The ramifications of Closures to popular programming languages are still in its infancy even though the concept has been around for decades (adding them to Java has been a debate for years). It will be extremely interesting to see how the wide spread adoption of Closures will ultimately impact our coding habits in the near future.

Will Python ever be taken seriously?

I’ve been writing Python for almost a year now and have been pleasantly surprised by the language. Even though Python has been around for a few decades now it still hasn’t been widely accepted as a production grade programming language.

Its primary use is still in the online world where it has developed a large rivalry with Ruby. It’s been my observation that Ruby wins this match up in the eyes of developers about 80% of the time. In the embedded world the use of Python is almost unheard of and I’ve personally never seen it used exclusively for a full stand alone application.

The language is aesthetically pleasing, requires less typing, and is fantastic for rapid prototyping. It definitely has its quarks and drawbacks but what language doesn’t? When I mention to other developers that I’m currently writing in Python the most common reaction is disbelief. The language just isn’t taken seriously. Why?

PyQt Charts

The Qt framework doesn’t contain any useful charting widgets. There are a few python charting modules out there, most notably PyQwt. As a PyQt developer I didn’t find this module very useful. What I really wanted was a charting widget that used QWidget as its base class. At the time of this post I couldn’t find anything so I wrote my own based on the google charts module.

What I love about this snippet is that it mimics the C++ template functionality in Python. The QChart function below abstracts the PyQtChart class definition and any pygooglechart object can be instantiated as a QWidget.

__author__	  = "Christopher Piekarski"
__email__	  = "@c_piekarski"

import time, os, sys
from PyQt4 import QtCore, QtGui
from pygooglechart import *

def QChart(parent, type, **kwargs):
	class PyQtChart(type, QtGui.QWidget):
		def __init__(self, parent, **kwargs):
			QtGui.QWidget.__init__(self, parent, **kwargs)
			type.__init__(self, kwargs["size"].width(), kwargs["size"].height())
			self.pix = QtGui.QPixmap()

		def download(self):
			file = "./%f.png" % time.time()
			type.download(self, file)
			self.pix.load(file)

		def paintEvent(self, event):
			p = QtGui.QPainter(self)
			p.drawPixmap(0,0,self.pix)
			super(PyQtChart, self).paintEvent(event)

	return PyQtChart(parent, **kwargs)

class MainWindow(QtGui.QMainWindow):
	def __init__(self, **kwargs):
		super(QtGui.QMainWindow, self).__init__(**kwargs)

		t = QChart(self, PieChart3D, size=QtCore.QSize(250,100))
		t.add_data([10,20])
		t.set_pie_labels(['Hello', 'World'])
		t.download()

if __name__ == '__main__':
	app = QtGui.QApplication(sys.argv)
	app.setApplicationName("PyQt Charting Demo")
	app.setQuitOnLastWindowClosed(True)

	scaledSize = QtCore.QSize(500,500)
	window = MainWindow(size=scaledSize)
	window.setWindowTitle("PyQt Charting Demo")
	window.show()

	sys.exit(app.exec_())

Getting the System IP Address with Python

The default Python 2.7.1 distribution has very limited functionality for getting the system IP. If your not willing to install an external module then you really only have one choice… open a socket, establish a connection, and get the active IP from the socket name tuple.

import socket
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.connect(('immersivelabs.com', 0))
s.getsockname()[0]

Although this solution works, it has a few issues. The biggest draw back is that it only returns one of the systems IP addresses. Fortunately, there is a great module called pynetinfo. Install the module using easy_install and then wrap its functionality with the following methods.

import netinfo

def getSystemIfs():
    return netinfo.list_active_devs()

def getSystemIps():
    """ will not return the localhost one """
    IPs = []
    for interface in getSystemIfs():
        if not interface.startswith('lo'):
            ip = netinfo.get_ip(interface)
            IPs.append(ip)
    return IPs

def getIPString():
    """ return comma delimited string of all the system IPs"""
    return ",".join(getSystemIps())

You can now easily get a list or a string containing all the active IP addresses on the system (POSIX Only).