Category Archives: Code

Unit Testing & Benchmarking – Google Go

Writing unit level tests and benchmarks are a critical aspect to ensuring software remaining reliable and performing as expected as a project grows. With many languages this requires the implementer to go out of their way to create a lot of special code, code that rarely gets updated as the unit properties change. Go makes this super easy and I am very impressed with the built in support for both of these operations. As an example, let’s look at a unit level test and benchmark for two different rot13 methods. Below are two methods for calculating a rot13 translation, one uses a look up table and the other calculates each individual value using the mod operator.

Code:

// rot13(rot13(x)) = x
// ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz
// NOPQRSTUVWXYZABCDEFGHIJKLMnopqrstuvwxyzabcdefghijklm

var table = map[byte]byte{'A':'N','B':'O','C':'P','D':'Q','E':'R','F':'S','G':'T','H':'U','I':'V','J':'W',
'K':'X','L':'Y','M':'Z','N':'A','O':'B','P':'C','Q':'D','R':'E','S':'F','T':'G','U':'H','V':
'I','W':'J','X':'K','Y':'L','Z':'M','a':'n','b':'o','c':'p','d':'q','e':'r','f':'s','g':'t',
'h':'u','i':'v','j':'w','k':'x','l':'y','m':'z','n':'a','o':'b','p':'c','q':'d','r':'e','s':
'f','t':'g','u':'h','v':'i','w':'j','x':'k','y':'l','z':'m'} 

func Rot13Table(input []byte) {
	for i, c := range input {
		input[i] = table[c]
	}
}

func Rot13Mod(input []byte) {
        for i, c := range input {
                base := byte(0)
                if c >= 'a' && c <= 'z' {
                        base = 'a'
                }else if c >= 'A' && c <= 'Z' {
                        base = 'A'
                }
                if base != 0 {
                        input[i] = byte((c - base + 13) % 26 + base)
                }
        }
}

Using Go’s built in approach for writing tests and benchmarks we can write a *_test.go file for ensuring our methods are producing the correct result and for profiling their performance. In Go, any method that starts with ‘Test’ or ‘Benchmark’ in a _test.go file are automatically compiled and executed when the ‘gotest’ command is run.

Code:

import (
	"testing"
	"fmt"
)

func BenchmarkRot13Mod(b *testing.B){
	translate := "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"

	temp := []byte(translate)
	for i:=0; i < b.N; i++ {
		Rot13Mod(temp)
	}
}

func BenchmarkRot13Table(b *testing.B){
	translate := "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz"

	temp := []byte(translate)
	for i:=0; i < b.N; i++ {
		Rot13Table(temp)
	}
}

The above (partial) snippet contains two benchmark tests, one for testing the table version and one for the mod operator approach. To run the test, simply execute “gotest -v -bench=.” within the unit and wait for the result. Each benchmark is looped b.N times and is repeated until Go acquires a high enough confidence in the value.

myHttp.BenchmarkRot13Mod 5000000 397 ns/op
myHttp.BenchmarkRot13Table 500000 3761 ns/op

Based on the above results, we can clearly see that the look up table approach is about 10x slower than the mod operator. Having this information available to future developers is highly desirable as it allows them to measure the impact of a unit level change and make an informed decision on which algorithm to use for a solution.

Expanding The Toolbox – Google Go

I’ve decided to experiment with 12 new programming languages or frameworks over the next year. Just like spoken languages, mastery of a coding language can take years even with daily use. With just one month per language I hope to develop a basic understanding of the languages strengths and weaknesses. Languages are tools that help us solve problems and having a broad perspective of what’s in the toolbox can simplify our solutions. Lots of problems can be solved with C, but why use it for the subset that Python can solve faster and more reliably?

All programming languages have a unique feel. Python fells concise and lucid whereas C++ feels bulky and ridged. For August’s language I picked Google’s Go. I selected Go because it supposedly has the feel of a dynamic interpreted language with the speed and type safety of a compiled language.

To get a feel for the syntax I wrote a simple closure that returns two functions, one for adding a list of integers to the sum and one for calculating the average. As you can see, Go certainly has the feel of a scripting language but requires variable type declaration. Like Python it also allows for returning multiple results from a method.

// Note: WordPress doesn't have a 'Go' syntax highlight yet so I'm using javascript
package main

import (
	"fmt"
)

//Closure that returns a sum and average method
func stats() (func(a []int) int, func() float64){
	sum := 0
	entries := 0
	log := func(){
		fmt.Printf("sum: %d, entries: %d\n",sum,entries)
	}
	return func(l []int) int {
		defer log()
		for i:=0;i<len(l);i++ {
			sum += l[i]
			entries += 1
		}
		return sum
	},
	func() float64 {
		return float64(sum)/float64(entries)
	}
}

func main() {
	sum,avg := stats()
	s := sum([]int{1,2,3})
	s = sum([]int{1,1,1,4,1})
	fmt.Printf("sum: %d avg: %f\n", s, avg())
}

As a long time C programmer I initially found the reversed type and declaration syntax to be awkward, but after writing a few hundred lines it actually flows nicely. Rob Pike put together a great three day tutorial that you can access online or in the /go/doc directory. For more information on the syntax check out the day one slides. By far the most compelling feature of Go is the built in support for concurrency. As someone who has done lots of multi-threaded and multi-process development I believe the Go development team did a fantastic job at making it a fundamentally ingrained principle. To illustrate this feature I wrote a concurrent Pell Number generator below.

package pell

// Pell Numbers
// 0, 1, 2, 5, 12, 29, 70, 169, 408, 985, 2378
// P0 = 0, P1 = 1, PN = 2PN-1 + PN-2

// Day 3: Do not communicate by sharing memory.
// Instead, share memory by communicating.

func Pell(ch chan<- uint64) {
	var fn_1, fn_2 uint64 = 1, 0
	ch <- 0
	ch <- 1
	for {
		fn_2, fn_1 = fn_1, ((2*fn_1) + fn_2)
		ch <- fn_1
	}
}

There is nothing special about the above Pell routine. Its parameter is a communication pipe (called a channel in Go) that guarantees synchronization because it is a blocking write. To start the thread you use the “go” keyword and read data from the same channel passed to the routine. Upon a read it unblocks the thread which then calculates the next number in the sequence.

package main

import ( "fmt"
	 "flag"
	 "pell"
)

func main() {
	c := make(chan uint64)
	var iterations *int = flag.Int("i", 10, "number of iterations to run")
	flag.Parse()

	go pell.Pell(c)
	for i:=0;i < *iterations; i++ {
		var num uint64 = <-c
		fmt.Println(num)
	}

}

From my perspective the one major red flag is the lack of support for object oriented design. Although the ability to add a method to a struct or any data type is very cool I don’t yet see how it can fully replace the concept of a class. I added a small miscellaneous Go repository on gitHub that anyone interested in some social coding can fork and modify at will.

Python Generators

Generators are an easy-to-use tool in Python for making any object method or class an iterator. I’ve found them to be a very useful development tool and wanted to briefly expand on my previous PyQt Charts post.

The simplest generator is an object method that uses the “yield” keyword. Yield is just like return only the instruction counter is persistent in the method. In the example below, a generator method is created to return characters of a string one at a time in reverse order. A more real world example would be making a read socket a generator that returns packet data.

def reversed(s):
    index = len(s)
    while index >= 0:
        yield s[index]
        index = index - 1

revGen = reversed("Android")
for c in revGen:
     print c

The second and more useful approach is to let classes be generators. This is my preferred method because OOD is always better :-). Every class definition can overload the __iter__ and next methods. When using the class approach the StopIteration exception is responsible for terminating the iteration.

class QCharts(object):
	def __init__(self, **kwargs):
		self.charts = []
		self.index = 0
		self.len = 0

	def __iter__(self):
		self.index = 0
		return self

	def next(self):
		if self.index == self.len:
			raise StopIteration
		self.index = self.index + 1
		return self.charts[self.index-1]

	def add(self, c):
		self.charts.append(c)
		self.len = len(self.charts)

Now, using this iterator class we can loop through a series of QWidget based charts. Below is a slightly modified MainWindow class from the PyQt Charts post.


from PyQt4 import QtCore, QtGui
from pygooglechart import PieChart3D

class QChart(parent, type, **kwargs):...
class QCharts(object):...	

class MainWindow(QtGui.QMainWindow):
	def __init__(self, **kwargs):
		super(QtGui.QMainWindow, self).__init__(**kwargs)
		self.charts = QCharts()

	def addPieChart(self):
		i = self.charts.len
		t = QChart(self, PieChart3D, pos=QtCore.QPoint(i*75,i*75), size=QtCore.QSize(250,100))
		t.add_data([(20*i)+10,(i*10)+20])
		t.set_pie_labels(['Hello', 'World'])
		t.download()
		self.charts.add(t)

	def debug(self):
		for x in self.charts:
			print x

if __name__ == '__main__':
	app = QtGui.QApplication(sys.argv)
	app.setApplicationName("Blank PyQt Demo")
	app.setQuitOnLastWindowClosed(True)

	scaledSize = QtCore.QSize(scale.width(500),scale.height(500))
	window = MainWindow(size=scaledSize)
	window.addPieChart()
	window.addPieChart()
	window.addPieChart()
	window.debug()
	window.setWindowTitle("Immersive Blank PyQt Demo")
	window.show()

	sys.exit(app.exec_())

For more information on generators check out this great write up here. Another great feature of generators is that they can be used in generator expressions. These expressions look like this:

#extract the even numbers from the series
ints = [0,1,2,3,4,5,6,7,8,9,10]
even = [x for x in ints if x % 2 == 0 ]

#get the directory names from glob
import glob
def dirs(p):
     return [x for x in glob.iglob(os.path.join( p, '*')) if os.path.isdir(x) ]
dirs("/")

JavaScript & Python Closures

Closures are generally considered to be JavaScripts most advanced and useful feature, and hence. As a C++ coder I like to think of Closures as a method with a privately malloc’d stack that is persistent across multiple invocations. As a Python coder I like to think of Closures as a method object with a preserved variable environment. Regardless of how you want to think about them, Closures allow us to redefine how we approach coding up methods.

Take the following simple JS example. Here, we are creating a very simple incrementer. The method inc() takes a starting value and returns a method that simply adds one to the start value (but not until its actually executed in the loop below).

function inc(start){
	var i = start;
	return function() { i = i + 1; document.write(i); }
}

addOne = inc(0)
for (i = 0; i < 10; ++i){
	addOne();
}
// prints 12345678910

I find this feature of the language very cool. We can also do the same thing with Python.

#Works with Python 3.x only
def inc(start):
    y = start
    def adder():
        nonlocal y
        y += 1
        print y
        return y
    return adder
 
addOne = inc(0)
x = 0
for x in range(10):
    addOne()
#prints 12345678910

The ramifications of Closures to popular programming languages are still in its infancy even though the concept has been around for decades (adding them to Java has been a debate for years). It will be extremely interesting to see how the wide spread adoption of Closures will ultimately impact our coding habits in the near future.

Will Python ever be taken seriously?

I’ve been writing Python for almost a year now and have been pleasantly surprised by the language. Even though Python has been around for a few decades now it still hasn’t been widely accepted as a production grade programming language.

Its primary use is still in the online world where it has developed a large rivalry with Ruby. It’s been my observation that Ruby wins this match up in the eyes of developers about 80% of the time. In the embedded world the use of Python is almost unheard of and I’ve personally never seen it used exclusively for a full stand alone application.

The language is aesthetically pleasing, requires less typing, and is fantastic for rapid prototyping. It definitely has its quarks and drawbacks but what language doesn’t? When I mention to other developers that I’m currently writing in Python the most common reaction is disbelief. The language just isn’t taken seriously. Why?

PyQt Charts

The Qt framework doesn’t contain any useful charting widgets. There are a few python charting modules out there, most notably PyQwt. As a PyQt developer I didn’t find this module very useful. What I really wanted was a charting widget that used QWidget as its base class. At the time of this post I couldn’t find anything so I wrote my own based on the google charts module.

What I love about this snippet is that it mimics the C++ template functionality in Python. The QChart function below abstracts the PyQtChart class definition and any pygooglechart object can be instantiated as a QWidget.

__author__	  = "Christopher Piekarski"
__email__	  = "@c_piekarski"

import time, os, sys
from PyQt4 import QtCore, QtGui
from pygooglechart import *

def QChart(parent, type, **kwargs):
	class PyQtChart(type, QtGui.QWidget):
		def __init__(self, parent, **kwargs):
			QtGui.QWidget.__init__(self, parent, **kwargs)
			type.__init__(self, kwargs["size"].width(), kwargs["size"].height())
			self.pix = QtGui.QPixmap()

		def download(self):
			file = "./%f.png" % time.time()
			type.download(self, file)
			self.pix.load(file)

		def paintEvent(self, event):
			p = QtGui.QPainter(self)
			p.drawPixmap(0,0,self.pix)
			super(PyQtChart, self).paintEvent(event)

	return PyQtChart(parent, **kwargs)

class MainWindow(QtGui.QMainWindow):
	def __init__(self, **kwargs):
		super(QtGui.QMainWindow, self).__init__(**kwargs)

		t = QChart(self, PieChart3D, size=QtCore.QSize(250,100))
		t.add_data([10,20])
		t.set_pie_labels(['Hello', 'World'])
		t.download()

if __name__ == '__main__':
	app = QtGui.QApplication(sys.argv)
	app.setApplicationName("PyQt Charting Demo")
	app.setQuitOnLastWindowClosed(True)

	scaledSize = QtCore.QSize(500,500)
	window = MainWindow(size=scaledSize)
	window.setWindowTitle("PyQt Charting Demo")
	window.show()

	sys.exit(app.exec_())

Web Apps – The More Custom The Better

I’m a big fan of the WebKit Qt framework integration. It will without a doubt change the face of web based content delivery over the next few decades (previous post). By far the most powerful aspect of this integration is the ability to create custom JavaScript APIs for hybrid web based applications in a browser.

Put simply, browsers can customize the functionally they offer by implementing features into the browser that can be accessed by web content via JavaScript.  Web apps that exercise this functionally must be written to do so, making them “custom” web apps.  Meaning that an app written to utilize features defined in certain browsers will not work without modification in Explorer, Firefox, or Chrome. For example, when GE integrates a web browser into their next generation refrigerators, it would be beneficial to expose an API that allowed an app to check the inside temperature or level of the ice tray.

I’ve been in many debates with other engineers about this option.  Not one engineer has ever agreed with me right off the bat.  Most oppose doing it as it limits were an app can be run and defeats the purpose of web content.  The “purpose” being mass distribution and play-ability on many different standardize browsers.  Most don’t understand why you’d create a web app that can only be executed on a limited number of browsers.  Many also argue that it burdens app developers with having to learn yet another JavaScript API.  Regardless of the objection, this option is widely opposed and not fully understood by most engineers.

The engineering objections are easy to argue against.  First of all, there is no such thing as a standardized browser.  Despite standardization efforts from W3C,  the JavaScript language is a mess.  The big three, IE, Firefox, and Chrome all use a different JavaScript engine and have varying degrees of standardization completeness (some even have features not covered by W3C).  If you want your website to work across the big 3 you already have to add additional code.  I know its a stretch, but this is already a form of web app customization.

The real value add of customized web apps is from a business perspective.  I come from the embedded device world where the number of units sold is the driving factor (compared to the number of hits). By forcing apps to write to your API you are ensuring a certain level of stickiness.  It can be a competitive advantage to limit the devices an app can run on by exposing custom functionality through the browser.  GE won’t want its ice tray level app to run on LG with no modification for the same reason Apple does’t want its apps to run on Android without a rewrite.

Adding a custom JavaScript API to your Qt WebKit browser is easy.  Nokia has a great blog post on its forum here. To illustrate its simplicity I’ve included my own snippet below.

Any C++ object can be complied into the browser and exposed to the JS engine.

class FakeObject
{
public:
	FakeObject(QObject *parent);
public slots:
	posterInternalCommand(QString command, QString data);
signals:
	fakeSignal();
}

FakeObject::FakeObject(QObject *parent)
{

}

bool FakeObject::postInternalCommand(QString command, QString data)
{
	// Do something with data
	return True;
}

FakeObject::emitSignalTest()
{
	emit fakeSignal();
}

After instantiation the object is exposed to the JS engine using a QWebFrame method.

FakeObject* m_pfake = new FakeObject(webView);
webView->page()->mainFrame()->addToJavaScriptWindowObject(QString("fake"),m_pFake);

The exposed object can now be acted on via JavaScript. Public slots are methods that can be called directly from JS. Signals can also be connected to local JS methods making for more asynchronous functionality.

b = window.fake.postInternalCommand("fakeCommand", "fakeData")

function fakeCallBack(){
	x = x + 1
}

window.fake.fakeSignal.connect(fakeCallBack)

Qt & WebKit Browsers

I don’t recall if it was Nokia or Trolltech who originally decided to add a WebKit widget to Qt. Regardless, its a fantastic addition to the framework and the ramifications to the world are still not yet fully known.

We’re all familiar with web browsers primarily on our PCs and mobile phones. That is going to change… in a big way. Qt has made it simple for development teams to create custom browsers that can run anywhere and inside any application. As a result we’re going to start seeing web browser in our cars, kitchen appliances, digital picture frames, remote controls, and even exercise equipment.

Below you will find an example I wrote using the Qt WebKit support. Its a very simple “custom” browser that allows an application to instantiate the object and render any HTML.

Header file (wijibrowser.h):

#include <QtGui>
#include <QWebView>

class WijiBrowser : public QMainWindow
{
    Q_OBJECT

public:
    WijiBrowser();
    ~WijiBrowser();

    bool loadUrl(QString name, QString url);
    bool loadHTML(QString name, QString html);
    QString contentName() { return m_name; }
    QString url() { return (view ? view->url().toString() : QString("none")); }

protected slots:
    void finishLoading(bool ok);

private:
    QString m_name;
    QWebView *view;
};

CPP file (wijibrowser.cpp):

#include <QtGui>
#include <QtWebKit>
#include "wijibrowser.h"

WijiBrowser::WijiBrowser()
{
    m_name = "unknown";

    QNetworkProxyFactory::setUseSystemConfiguration(true);

    view = new QWebView(this);
    view->load(QUrl("http://www.immersivelabs.com"));
    connect(view, SIGNAL(loadFinished(bool)), SLOT(finishLoading(bool)));

    setCentralWidget(view);
    setUnifiedTitleAndToolBarOnMac(true);
}

WijiBrowser::~WijiBrowser()
{
	delete view;
}

void WijiBrowser::finishLoading(bool ok)
{
	QString status = (ok ? "OK" : "Failed");
	QString url = view->url().toString();
	QString msg = QString("finishLoading %1 - %2").arg(url).arg(status);
	qDebug(msg);
}

bool WijiBrowser::loadUrl(QString name, QString url)
{
	m_name = name;
    view->load(QUrl(url));
}

bool WijiBrowser::loadHTML(QString name, QString html)
{
	m_name = name;
	view->setHtml(html);
}

And here is a simple main.cpp to launch it:

#include <QtGui>
#include "wijibrowser.h"

int main(int argc, char * argv[])
{
    QApplication app(argc, argv);
    WijiBrowser browser;
    browser.show();
    return app.exec();
}

Getting the System IP Address with Python

The default Python 2.7.1 distribution has very limited functionality for getting the system IP. If your not willing to install an external module then you really only have one choice… open a socket, establish a connection, and get the active IP from the socket name tuple.

import socket
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.connect(('immersivelabs.com', 0))
s.getsockname()[0]

Although this solution works, it has a few issues. The biggest draw back is that it only returns one of the systems IP addresses. Fortunately, there is a great module called pynetinfo. Install the module using easy_install and then wrap its functionality with the following methods.

import netinfo

def getSystemIfs():
    return netinfo.list_active_devs()

def getSystemIps():
    """ will not return the localhost one """
    IPs = []
    for interface in getSystemIfs():
        if not interface.startswith('lo'):
            ip = netinfo.get_ip(interface)
            IPs.append(ip)
    return IPs

def getIPString():
    """ return comma delimited string of all the system IPs"""
    return ",".join(getSystemIps())

You can now easily get a list or a string containing all the active IP addresses on the system (POSIX Only).

Thrift RPC Framework

At TechStars, Alessio Signorini and I had the opportunity to experiment with Facebook’s Thrift remote procedure call (RPC) framework. I was thoroughly impressed with the framework and feel it takes distributed computing to whole new level. Every developer regularly working on both large and small client/server architectures must be familiar with this tool.

Developers can easily specify data types (parameters) and server interface definitions (API) in a plain text file. Thrift uses this text file to create programming language independent client and server stub files. What makes this so powerful, is that system architects can now choose the appropriate programing language for each area of the system.

Not all programing languages are created equal and no one language contains just positive attributes. It is often the case that the ideal language for a server is Java or C++, but the client would best be written in Python. Thrift allows you to easily choose the best language for the job without having to reinvent the communication protocol every-time you change the language.

This framework is a game changer for software developers and system architects. If you haven’t already done so I strongly encourage you to start playing with it. Below is an example thrift file that defines several data types and one service. Once this interface is defined it can easily be converted into stubs for any language which keeps the interface API separate from the implementation. Read the whitepaper here.

#!/usr/local/bin/thrift --gen cpp:pure_enums --gen php

namespace cpp displaymanager

enum pushResult
{
  OK,
  BUSY,
  INVALID
}

struct pushUrl
{
  1:  string name,
  2:  string url
}

struct pushHtml
{
  1:   string name,
  2:   string data
}

service displaymanager
{
  pushResult fetchUrl(1: pushUrl url);
  pushResult postHtml(1: pushHtml html);
  string	 whatsLoaded();
  string	 currentUrl();
}