Tag Archives: Python

Python code.

Thrift RPC Framework

At TechStars, Alessio Signorini and I had the opportunity to experiment with Facebook’s Thrift remote procedure call (RPC) framework. I was thoroughly impressed with the framework and feel it takes distributed computing to whole new level. Every developer regularly working on both large and small client/server architectures must be familiar with this tool.

Developers can easily specify data types (parameters) and server interface definitions (API) in a plain text file. Thrift uses this text file to create programming language independent client and server stub files. What makes this so powerful, is that system architects can now choose the appropriate programing language for each area of the system.

Not all programing languages are created equal and no one language contains just positive attributes. It is often the case that the ideal language for a server is Java or C++, but the client would best be written in Python. Thrift allows you to easily choose the best language for the job without having to reinvent the communication protocol every-time you change the language.

This framework is a game changer for software developers and system architects. If you haven’t already done so I strongly encourage you to start playing with it. Below is an example thrift file that defines several data types and one service. Once this interface is defined it can easily be converted into stubs for any language which keeps the interface API separate from the implementation. Read the whitepaper here.

#!/usr/local/bin/thrift --gen cpp:pure_enums --gen php

namespace cpp displaymanager

enum pushResult
{
  OK,
  BUSY,
  INVALID
}

struct pushUrl
{
  1:  string name,
  2:  string url
}

struct pushHtml
{
  1:   string name,
  2:   string data
}

service displaymanager
{
  pushResult fetchUrl(1: pushUrl url);
  pushResult postHtml(1: pushHtml html);
  string	 whatsLoaded();
  string	 currentUrl();
}

Python – Love / Hate

I hate about Python because:

1) Really bad at managing threads.
2) All class properties require “self” keyword.
3) Inheritance is supported but can be difficult to invoke.
4) Private members are difficult to encapsulate.
5) More difficult to deploy then traditional compiled languages.
6) Default timer class is limiting.

I love about Python because:

1) Tuples.
2) Keyword parameters.
3) Everything is an object.
4) The “pass” keyword.
5) Generator objects.
6) Method objects.
7) Dynamic code insertion.
8) PEP8.

Python – No Enumerated Data Type?

In the embedded world where C/C++ are the predominantly used languages, the enum data type is heavily used. It’s quick and easy to use and understand for specifying flags and register values.  Not yet understanding that Python’s real strength is hash-able data types and that a switch statement didn’t exist, I found myself wanting to use traditional C/C++ enumerated data types when I first started learning Python.

Even though there is no “enum” keyword in Python, there are a few easy ways to create an enumerated data type. The following snippet uses the Python argument packer to create a new data type that is just a key value paired dictionary.

def enum(*args):
    enums = dict(zip(args, range(len(args))))
    return type('Enum', (), enums)

Thats it. This simple function takes a list of items and auto-completes the value. It can easily be used like the following.

alignment = enum("left","center","right")

def alignText(text, pos):
    if pos == alignment.left:
        pass

alignText("some text", alignment.left)

Another object oriented approach I came across is to overload the existing Python set( ) data type and override the __getattr__() method.

class Enum(set):
    def __getattr__(self, name):
        if name in self:
            return name
        raise AttributeError

    def __setattr__(self, name, value):
        pass

    def __delattr__(self, name):
        pass

Neither approach fully addresses all the functionality of the C/C++ enum type. The missing functionality is the ability to specify a value for an entry and having all sequential entries be incremented by one (below). However, both solutions do offer a quick enumerated data type that can be used in most scenarios.

enum alignment = {	LEFT,		// 0 
					CENTER=10,	// 10
					RIGHT		// 11 
		 		 };


PyMT Black Backdrop Widget

Over the past few PyMT applications I’ve found it very useful to have a black backdrop to place over the main application area to highlight a special area of the app. The basic widget below absorbs all the key events that would normally make it to the main app. Its easy to use, just make it visible and then raise the widgets you want above it.

class Backdrop(MTWidget):
    def __init__(self, **kwargs):
        super(Backdrop, self).__init__(**kwargs)
        self.style = {'bg-color': (0,0,0,.6), 'draw-background': 1}
        
    def on_touch_down(self, touch):
        if not self.collide_point(*touch.pos):
            return
        touch.grab(self)
        return True
    def on_touch_up(self, touch):
        if touch.grab_current != self:
            return
        return True
    def on_touch_move(self, touch):
        return
    
b = Backdrop(size=(1920,1080), pos=(0,0)) #full width of app window
b.hide()

PyMT Scatter Video Widget

For anyone contemplating writing a multi-touch application, PyMT can be a fun and easy way to see instant results. Its API offers several useful widgets that allows for fast prototyping. With that said, PyMT should not be used for a commercial or high performance applications, as it has some major limitations with its inheritance architecture and screen paint logic (plus it comes with a GPL license). Some of these issues have been addressed in the teams rewrite of PyMT called Kivy. Although Kivy appears promising, it is still in its infancy and will need a few more release iterations before being adopted by the masses.

The weakest PyMT widget is by far the video player. Its integration with gstreamer is not nearly as complete as PyQt and it doesn’t have full multi-touch support. To address the multi-touch issue I wrote the following code snippet for one of Immersive’s first early stage in store product browser. It adds a default picture when the video is at position 0 and superimposes a “play” button when paused. In addition it uses the MTScatter widget as its base class which allows the user to manipulate the video using full multi-touch gestures when playing.

class scatVid(MTScatter):
    def __init__(self, filename, image, **kwargs):
        # image is a pre-constructed Image( ) object to be used at start of video
        super(scatVid, self).__init__(**kwargs)
        self.image = MTImage(image)
        self.play = MTImage(Image('./play_button.png'))
        self._posXY = None
        self.video = MTVideo(filename=filename, eos='loop')
        self.video.hide_controls()
        self.video.player.position = 10
        self.size=self.video.size
        connect(self.video, 'on_resize', self, 'size')
        connect(self.video, 'on_resize', self.image, 'size')
        connect(self.video, 'on_resize', self.play, 'size')

        self.add_widget(self.video)
        self.add_widget(self.image)
        self.add_widget(self.play)
        self._state = ""

    def stop(self):
        self.video.player.stop()

    def play(self):
        self.video.player.play()

    def hide(self):
        self.video.player.stop()
        super(scatVid, self).hide()

    def show(self):
        self.video.player.load()
        super(scatVid, self).show()

    def _get_state(self):
        return self.video.player.state

    def _get_posXY(self):
        return self._posXY
    def _set_posXY(self, x, y):
        print "setting posXY: %dx%d" %(x,y)
        self._posXY=(x,y)
        self.pos=(x,y)

    def draw(self):
        super(scatVid, self).draw()
        if self.state != "playing":
            if self.video.player.position < 2:
                self.image.show()
            else:
                self.play.show()
        else:
            self.image.hide()
            self.play.hide()

    state = property(lambda self: self._get_state(),
            doc='Get the video playing status')

To see the above code in action look here. You can use any play button piece of artwork, but for your convince here is the one I used.

Super Easy Python JSON Client & Server

Visit the Redux page for this post to get most recent information on jscoket package.

While at Immersive Labs, Alessio Signorini and I frequently found ourselves having to write custom mini-servers for interactive content. To help minimize the differences between the custom content we devised two super simple object oriented JSON client and server objects. Using this module you can easily create a custom JSON messaging server by overloading just one method. The latest code plus examples can be found on GitHub here. In addition, official releases can be found on PyPi and installed using “easy_install jsocket”.

The JsonSocket class below is the base class for both the client and server objects. Its responsible for handling all the socket level send/receive operations. Each message contains two parts, a header and a payload. The header is nothing more then the length of the payload and the payload is just a JSON object.

#git tag: v1.1-pypi
""" contains a json object message passing server and client """

import json
import socket
import struct
import logging

logger = logging.getLogger("jsonSocket")
logger.setLevel(logging.DEBUG)
FORMAT = '[%(asctime)-15s][%(levelname)s][%(funcName)s] %(message)s'
logging.basicConfig(format=FORMAT)

class JsonSocket(object):
    def __init__(self, address='127.0.0.1', port=5489):
        self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        self.conn = self.socket
        self._timeout = None
        self._address = address
        self._port = port

    def sendObj(self, obj):
        msg = json.dumps(obj)
        if self.socket:
            frmt = "=%ds" % len(msg)
            packedMsg = struct.pack(frmt, msg)
            packedHdr = struct.pack('=I', len(packedMsg))

            self._send(packedHdr)
            self._send(packedMsg)

    def _send(self, msg):
        sent = 0
        while sent < len(msg):
            sent += self.conn.send(msg[sent:])

    def _read(self, size):
        data = ''
        while len(data) < size:
            dataTmp = self.conn.recv(size-len(data))
            data += dataTmp
            if dataTmp == '':
                raise RuntimeError("socket connection broken")
        return data

    def _msgLength(self):
        d = self._read(4)
        s = struct.unpack('=I', d)
        return s[0]

    def readObj(self):
        size = self._msgLength()
        data = self._read(size)
        frmt = "=%ds" % size
        msg = struct.unpack(frmt,data)
        return json.loads(msg[0])

    def close(self):
        logger.debug("closing main socket")
        self._closeSocket()
        if self.socket is not self.conn:
            logger.debug("closing connection socket")
            self._closeConnection()

    def _closeSocket(self):
        self.socket.close()

    def _closeConnection(self):
        self.conn.close()

    def _get_timeout(self):
        return self._timeout

    def _set_timeout(self, timeout):
        self._timeout = timeout
        self.socket.settimeout(timeout)

    def _get_address(self):
        return self._address

    def _set_address(self, address):
        pass

    def _get_port(self):
        return self._port

    def _set_port(self, port):
        pass

    timeout = property(_get_timeout, _set_timeout,doc='Get/set the socket timeout')
    address = property(_get_address, _set_address,doc='read only property socket address')
    port = property(_get_port, _set_port,doc='read only property socket port')

The JsonServer class defined below inherits the above JsonSocket and makes available many of the standard server operations such as listen, bind and accept. You can invoke this class directly if you want to add a JsonServer to an already existing thread. However, the preferred method is to inherit the ThreadedServer class described below.

#git tag: v1.1-pypi
class JsonServer(JsonSocket):
    def __init__(self, address='127.0.0.1', port=5489):
        super(JsonServer, self).__init__(address, port)
        self._bind()

    def _bind(self):
        self.socket.bind( (self.address,self.port) )

    def _listen(self):
        self.socket.listen(1)

    def _accept(self):
        return self.socket.accept()

    def acceptConnection(self):
        self._listen()

        self.conn, addr = self._accept()
        self.conn.settimeout(self.timeout)
        logger.debug("connection accepted, conn socket (%s,%d)" % (addr[0],addr[1]))

Just like the JsonServer class above, the JsonClient also inherits the JsonSocket base class. For simplicity, and thanks to the functionality encapsulation of JsonSocket, the client only needs to support one connect method.

#git tag: v1.1-pypi
class JsonClient(JsonSocket):
    def __init__(self, address='127.0.0.1', port=5489):
        super(JsonClient, self).__init__(address, port)

    def connect(self):
        for i in range(10):
            try:
                self.socket.connect( (self.address, self.port) )
            except socket.error as msg:
                logger.error("SockThread Error: %s" % msg)
                time.sleep(3)
                continue
            logger.info("...Socket Connected")
            return True
        return False

In order for the JsonServer to be useful a thread must be attached to it. The following ThreadedServer class inherits both JsonSocket and Python’s thread class. By inheriting both of these classes we can combine all of the server functionality into one class, making it unnecessary to change any of the underlying message passing functionality. Currently this implementation accepts only one connection per thread. Notice that the “private” _processMessage method must be implemented by another inheriting class.

#git tag: v1.1-pypi
""" A threaded server based on a JsonServer object in the jsonSocket module """
import jsonSocket
import threading
import socket

import logging

logger = logging.getLogger("threadedServer")
logger.setLevel(logging.DEBUG)
FORMAT = '[%(asctime)-15s][%(levelname)s][%(funcName)s] %(message)s'
logging.basicConfig(format=FORMAT)

class ThreadedServer(threading.Thread, jsonSocket.JsonServer):
	def __init__(self, **kwargs):
		threading.Thread.__init__(self)
		jsonSocket.JsonServer.__init__(self)
		self._isAlive = False

	def _processMessage(self, obj):
		""" virtual method """
		pass

	def run(self):
		while self._isAlive:
			try:
				self.acceptConnection()
			except socket.timeout as e:
				logger.debug("socket.timeout: %s" % e)
				continue
			except Exception as e:
				logger.exception(e)
				continue

			while self._isAlive:
				try:
					obj = self.readObj()
					self._processMessage(obj)
				except socket.timeout as e:
					logger.debug("socket.timeout: %s" % e)
					continue
				except Exception as e:
					logger.exception(e)
					self._closeConnection()
					break

	def start(self):
		self._isAlive = True
		super(ThreadedServer, self).start()

	def stop(self):
		""" The life of the dead is in the memory of the living """
		self._isAlive = False

Finally, devising a custom server is as simple as creating a new class which inherits ThreadedServer and implementing the _processMessage(obj) method. To implement a client just instantiate a JsonClient object, execute connect(), and pass messages back and forth using send/readObj(obj).

#git tag: v1.1-pypi
import jsonSocket

class MyServer(jsonSocket.ThreadedServer):
    def __init__(self):
        super(MyServer, self).__init__()
        self.timeout = 2.0

    def _processMessage(self, obj):
        """ virtual method """
        if obj != '':
            if obj['message'] == "new connection":
                pass

if __name__ == "__main__":
    c = MyServer()
    c.start()