Wednesday, December 25, 2013

Development Server: Automatic Reload on Code Change

There are actually many ways of automatically reloading code when it is modified.  Some are platform/language specific and some are not, although they do depend on certain common behaviors.  This is one I'm using to develop my Python/Gunicorn application and it isn't specific to Python or Gunicorn; however it does require that you have inotify-tools installed, your server can run in the foreground and that it reloads the project when it receives a SIGHUP signal.




trap 'kill -TERM $RUNNING_PID 2> /dev/null; exit' SIGINT

while /bin/true; do
 echo "Starting '$SERVER'..."
 inotifywait -q --exclude '.*\.py[co]$' \
           -e modify -e close_write -e move \
           -e create -e delete \
           -r $WATCHED_DIR
 echo "Hupping '$SERVER'..."

You can then call it like this:

wrapper 'gunicorn project:main'

This will watch the current directory your in '.' and anytime a modification, creation, deletion or move occurs on any file in the current directory, inotify will issue the notification and stop waiting.  This will cause kill to send a SIGHUP to the the server forcing it to reload the project.  inotify will then wait on the next filesystem event.




There are a few variations on this theme which are fairly simple.

  • If you must kill and then restart the process in order to reload you can move the execution of $SERVER & into the while loop.  You should also change the HUP to a TERM in this case to make sure the process is terminated.
  • If you need to reload when anything changes in multiple directories you can just append the full list to inotifywait or generalize the wrapper and take the directories to watch as an argument.
  • If you want to or have to use something different from inotify-tools you can.  This same process should be usable by any of the file system event notification frameworks as long as the have a script that waits on events or allows you to write a script that waits on event.


Friday, April 19, 2013

Python str with custom truth values

I've run into multiple instances where I get a string from some external service like a config file or a database.  There are values that I want to treat as true and others as false for the purposes of logic in my code.  For example let's say I have a status string with a few possible values, some false and some true.
  1. "Yes", "on", "true"
  2. "No", "off", "false"
I can map the values to True or False in Python.  Now I just need a string I can configure what it's truth values are.

class BooleanString(str):
    def __nonzero__(self):
        return self.lower() in ('on', 'yes', 'true') 

bool(BooleanString("YES")) # return True
bool(BooleanString("some other value")) # returns False

The thing is I don't want to hard code the truth values in the class definition.  I want to pass them in to the creation of the derived class.

def mkboolstr(truth):
    def __nonzero__(self):
        return self.lower() in truth
    return type('BooleanString', (str, ), dict(__nonzero__=__nonzero__))

BooleanString = mkboolstr(('on', 'yes', 'true'))

bool(BooleanString("TRUE")) # return True
bool(BooleanString("some other value")) # returns False

And now we have a function which creates a BooleanString class using a parameterized string as the comparable truth.


Python Expressions: Merging Dictionaries

It's common in Python programming to need to merge 2 or more dictionaries together. 

The first idiom is using the dict constructor.  This idiom has it's limitations, however it will always work fine as long as the keys are all strings. Trying this with non-string keys will fail in Python 3.2 and later, and also fails in alternate Python implementations. The idiom itself is frowned upon.

d1 = dict(a=1, b=2, c=3)
d2 = dict(c=4, d=5, e=6)

# merging 2 dicts with the dict constructor
merged_dict = dict(d1, **d2)

# merging n dicts with the dict constructor
merged_dict = reduce(lambda a, b: dict(a, **b), (d1, d1))

There are also other choices which will work with any type of key. Unfortunately they require a tad more code.

# merging n dicts with a generator comprehension
merged_dict = dict(i for iterator in (d1, d2) for i in iterator.iteritems())

# merging n dicts with dict comprehension
merged_dict = {k:v for d in (d1, d2) for k, v in d.iteritems()}


I have left out dict.update because it is only usable in a statement, not an expression. It also modifies a dictionary which may not be what I want to do. You can compare:

def fn1(d1, d2):
   d3 = d1.copy()
   return d3

# vs

return dict(d1, **d2)

# or

return {k:v for d in (d1, d2) for k, v in d.iteritems()}

I prefer the expressions.

Thursday, April 18, 2013

Python Expressions: Dict Slicing

As anyone who comes from Perl can tell you hash slicing is useful.  Python dicts do not natively support slicing.  One of the issues is slicing in Python seems limited to defined ranges, rather than an ad-hoc collection of values.

Take heart! There is an expression that has the same effect in Python as Perl's hash slicing:

from operator import itemgetter

d = dict(a=1, b=2, c=3, d=4)
h, i, j, k = itemgetter('a', 'c', 'a', 'd')(d)
# h, i, j, k => 1, 3, 1, 4

Alternate solutions are usually more complicated expressions.
d = dict(a=1, b=2, c=3, d=4)

# conceptually more complicated expressions
h, i, j, k = (d[i] for i in ('a', 'c', 'a', 'd'))
h, i, j, k = map(d.get, ('a', 'c', 'a', 'd'))

itemgetter provides a more simple expression for extracting multiple ad-hoc values from a dictionary.