A small history dive

This used to be very visible and meant almost “do not perform SQL queries in a cycle”:

1
2
3

books = c.execute("SELECT id, title, author_id FROM books").fetchall()  # 1
for id, title, author_id in books:
    c.execute("SELECT full_name FROM authors WHERE id=?", [author_id])  # +N

With ORM and lazy loading this became a little bit less obvious:

1
2
3

books = Book.objects.filter(...)  # 1
for book in books:
    print(f"Book title: {book.title}, author: {book.author.full_name}")  # +N

With something so innocent as an attribute access making an SQL query, it’s much easier to miss it. Especially when this code spreads out, the ORM objects are passed to templates, which also have loops and sure they can do attribute access.

As project grows, as its database schema becomes more complicated, as your team grows too, this keeps adding up. And magic also adds up. One particular mention should be a GraphQL library, which resolves onto ORM automatically.

Back to the present

I tumbled on a couple of 1+Ns while reading a project code for an unrelated reason and it got me thinking – do I ever want Django to do that lazy loading stuff? And the answer was never. This was a misfeature for me, the need for such thing is quite circumstantial, usually when you load a list of things you need the same data about all of them, so it doesn’t make sense to lazy load extra data for each object separately. Either eager load or batch lazy load, the latter Django does not do.

So, anyway, if I don’t need this than I might as well prohibit it, which turned out to be quite easy to do:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

from django.db.models.query_utils import DeferredAttribute


def _DeferredAttribute_get(self, instance, cls=None):
    if instance is None:
        return self
    data = instance.__dict__
    field_name = self.field.attname
    if field_name in data:
        return data[field_name]

    # Raise an exception to prevent an SQL query
    attr = f"{instance.__class__.__name__}.{field_name}"
    message = f"Lazy fetching of {attr} may cause 1+N issue"
    raise LookupError(message)

DeferredAttribute.__get__ = _DeferredAttribute_get

This way 1+N will blow up instead. Great, we’ll catch it during tests. The thing is, however, if 1+Ns were passing our defences before they will probably continue now and this will explode in production. With this in mind, a flood guard and some explanations it transforms into:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

import logging
import os
from django.db.models.query_utils import DeferredAttribute

logger = logging.getLogger(__name__)
attrs_seen = set()


def _DeferredAttribute_get(self, instance, cls=None):
    from django.conf import settings  # monkeys go early, settings might not be available yet

    if instance is None:
        return self
    data = instance.__dict__
    field_name = self.field.attname

    # Normally this accessor won't be called if field_name is in __dict__,
    # we need this part so that DeferredAttribute descendants with __set__ play nice.
    if field_name in data:
        return data[field_name]

    # If it's not there already then prevent an SQL query or at least notify we are doing smth bad
    attr = f"{instance.__class__.__name__}.{field_name}"
    # Only trigger this check once per attr to not flood Sentry with identical messages
    if attr not in attrs_seen:
        attrs_seen.add(attr)
        message = f"Lazy fetching of {attr} may cause 1+N issue"
        # We stop in DEBUG mode and if inside tests but let production to proceed.
        # Using LookupError instead of AttributeError here to prevent higher level "handling" this.
        if settings.DEBUG or "PYTEST_CURRENT_TEST" in os.environ:
            raise LookupError(message)
        else:
            logger.exception(message)

    # Proceed normally
    return _DA_get_original.original(self, instance, cls)

_DA_get_original, DeferredAttribute.__get__ = DeferredAttribute.__get__, _DeferredAttribute_get

Which is ready to be used as is. Simply need to put or import it somewhere.

P.S. A small bonus – how I tried to make ChatGPT write this post for me. It was mostly failure :), but refactoring the code sample was done nicely.

Previously I described various ways AST could be used to alter Python language. Now I will concentrate on translation, and we will translate Python to JavaScript.

Translation

Unlike all Part 1 examples this one started from real need, the one as mundane as form validation:

Here we have a form with a validation error visible, we typically want this to be shown as soon as possible, so we implement these checks in browser in JavaScript. But we still need to check everything on the server to not let broken data into our database.

Usually we just duplicate logic and this works. Until we get many forms and all of them are like this:

When we arrive to this scale code duplication starts to cause problems duplication always does: some things are checked only in one place, some things are not updated accordingly and contradict each other. We show weird errors to our users and save broken data to our database.

In ideal world we wish to write every rule just once and then use it in both environments:

1
2
3
4
5

# Clean raw value
clean = lambda x: int(re.sub(r'\s+', '', x))

# Validate clean value
validate = lambda x: 1000 <= x < 100000000

So we naturally came to translation. We want to write validation in python, translate it to JavaScript and use on front-end automatically.

The Easy Part

Let’s start from something simple, like translating a plain lambda like this:

1

>>> translate(lambda v: v <= 100)

We will use AST to for translation, so first we need to get it for this lambda. It’s hard to get source for lambda to parse it, so we’ll use meta library to decompile its bytecode:

1
2
3

from meta.decompiler import decompile_func

tree = decompile_func(lambda v: v <= 100)

Once we do that we get a tree like this:

I simplified it to look prettier, but all significant aspects are still here.

The easiest way to render it to a new language is to start from leaves: numbers, variable names and operators mostly look the same in JavaScript. We will employ ast.NodeVisitor to walk the tree:

1
2
3
4
5
6
7
8
9

class Translator(ast.NodeVisitor):
    def visit_Num(self, node):
        node.js = str(node.n)

    def visit_Name(self, node):
        node.js = 'null' if node.id == 'None' else node.id

    def visit_LtE(self, node):
        node.js = '<='

After using this we come to:

When leaf nodes are done we can compile code for ones depending on them:

1
2
3
4
5
6

def visit_Compare(self, node):
    # Visit sub-nodes
    self.generic_visit(node)
    # Concatenate sub-node parts,
    # use parentheses to not worry about operator precedence
    node.js = '(%s %s %s)' % (node.left.js, node.op.js, node.right.js)

And we’ve got to:

We can continue going up the tree and finally get:

1
2
3

>>> translate(lambda v: v <= 100)
function (v) {
    return (v <= 100)}

Yay! We did it. But…

Complications

What about translating code like:

1

1 <= x < 10

There is no chained comparisons in JavaScript, the best we can do is translating this to:

1

(1 <= x) && (x < 10)

There is also no not in operator in JavaScript, combining all that Compare translation complicates to (not necessary to read and understand all this, just appreciate the complication):

1
2
3
4
5
6
7
8
9
10
11

def visit_Compare(self, node):
    self.generic_visit(node)

    ops = node.ops
    operands = [node.left] + node.comparators
    pairs = pairwise(operands)  # adjacent pairs of operands

    node.js = ' && '.join('%s(%s %s %s)' % \
        ('!' if isinstance(op, ast.NotIn) else '',
            l.js, op.js, r.js)
        for op, (l, r) in zip(ops, pairs))

Another complication is keyword arguments and star arguments. We can deal with them by complecting our code even more or by refusing to handle this case at all:

1
2
3
4

def visit_Call(self, node):
    assert node.kwargs is None \
        and node.starargs is None
    # ...

Now we are translating only subset of Python, but this should be okay since translation is not our goal, it’s code deduplication for validation rules. So as long as translator works for this particular use case we are fine.

Closures

We are still not done yet. We probably don’t want to write same validation rules over and over, so we will end with code like:

1
2
3
4

>>> up_to = lambda limit: lambda v: v <= limit
>>> translate(up_to(50))
function (v) {
    return (v <= limit)}

It’s easy to see that limit in JavaScript code refers to non-existing variable. To fix that we need to introspect values enclosed within a function, thankfully Python introspection has our back:

1
2
3

>>> u50 = up_to(50)
>>> u50.__closure__
(<cell at ...: int object ...>,)

Here the closure is a tuple containing single cell with an int object in it. Makes sense since we are enclosing 50. It’s fairly easy to get cell contents:

1
2
3

>>> [cell.cell_contents
      for cell in u50.__closure__]
[50]

It’s also easy to introspect free variables refering to enclosed values:

1
2

>>> u50.__code__.co_freevars
('limit',)

This co_freevars always have the same length and order as cells in closure, so we can pair them up and translate the whole thing to:

1
2
3
4
5
6

(function (){
    var limit = 50;

    return function (v) {
        return (v <= limit)}
}())

So we’ve got a closure. This highlights how we are not really translating functions, but closures, which are functions plus context.

More Closures

Unfortunately func.__closure__ doesn’t capture everything, any global names and built-ins are still uncovered. Let’s translate this one:

1

len50 = lambda s: len(s) <= 50

We can access function globals and built-ins namespaces with:

1
2
3
4
5

>>> len50.__globals__
{'len50': <function <lambda> at 0x7f705a92a848>, '__builtins__': <module '__builtin__' (built-in)>, 'dis': <module 'dis' from '/usr/lib/python2.7/dis.pyc'>, 'translate': <function js.translate>, 's': '\n<section>\n<pre><code class="python">>>> [cell.cell_contents\n...'

>>> __builtin__.__dict__
{'bytearray': <type 'bytearray'>, 'IndexError': <type 'exceptions.IndexError'>, 'all': <built-in function all>, 'vars': <built-in function vars>, 'SyntaxError': <type 'exceptions.SyntaxError'>, 'isinstance': <built-in function isinstance>, 'copyright': Copyright (c) 2001-2014 Python Software Foundatio...

So we only need to detect global names used by a function. We could have walked the AST, but then we would need to separate local and enclosed non-local names from global ones. The easier way is to introspect bytecode, which looks like this:

1
2

>>> len50.__code__.co_code
't\x00\x00|\x00\x00\x83\x01\x00d\x01\x00k\x01\x00S'

Just a binary string, not that pretty. But python standard library provides us dis module to handle that:

1
2
3
4
5
6
7

>>> dis.dis(len50)
1       0 LOAD_GLOBAL         0 (len)
        3 LOAD_FAST           0 (s)
        6 CALL_FUNCTION       1
        9 LOAD_CONST          1 (50)
       12 COMPARE_OP          1 (<=)
       15 RETURN_VALUE

The thing to see here is LOAD_GLOBAL instruction, it corresponds to global variable use. dis module only prints bytecode nicely, but doesn’t let us really in. So we go for another asset — byteplay:

1
2
3
4
5
6
7
8

from byteplay import Code, LOAD_GLOBAL

# Wrap code into byteplay object
code = Code.from_code(f.__code__)

# Iterate over instructions and collect names
names = {param for cmd, param in code.code
               if cmd == LOAD_GLOBAL}

So we collected all refered global names, we can translate them unless they are implemented in C. To handle that we can supply a dict like:

1
2
3
4
5

BUILTINS = {
    ...
    len: lambda s: s.length,
    ...
}

With code which doesn’t make sense in Python. However, it will make sense after translation:

1
2
3
4
5
6
7

>>> translate(lambda s: len(s) <= 50)
(function (){
    var len = function (s) {return s.length};

    return function (s) {
        return (len(s) <= 50)}
}())

Actually we could have used strings in BUILTINS dict, but with lambdas we can refer to other things in our implementations:

1
2
3
4
5
6

BUILTINS = {
    ...
    identity: lambda x: x,
    all: lambda seq: seq.every(identity),
    ...
}

Which enables us to make things like:

1
2
3
4
5
6
7
8
9
10
11
12
13

(function (){
    var all = (function () {
            var identity = function (x) {
                return x};

            return function (seq) {
                return seq.every(identity)}
        }()),
        len = function (s) {return s.length};

    return function (xs) {
        return ((len(xs) >= 1) && all(xs))}
}())

Here all() is defined as a nested closure. Also the whole thing starts to look complex. Fortunately we don’t need to write this, we don’t even need to read this, we only deal with python lambdas. And that one is just a single line for the mess above:

1

lambda xs: len(xs) >= 1 and all(xs)

Meeting Halfway

We’ve already come a long way, however, some things still won’t work:

1
2

# Does't work, it's s.trim() in JavaScript
lambda s: s.strip()

To fix this we will need to write shims for all python types and convert everything before passing to our function. This sounds like a lot of work, so went another way — just don’t write like that, write this way instead:

1
2

# Does work if "string" is in BUILTINS
lambda s: string.strip(s)

Look, system does many things to make everything work, now we can come along and finally meet halfway. This approach worked wonderfully for us, we managed to get practical solution with:

• no runtime,
• no shims,
• only 300 lines of code.

Compare this to full-blown python to js translator. Anyway let’s see what it have brought to us.

Demos

Here is how validation looks. It updates as we type and wholly executed in the browser:

And we don’t write any JavaScript to achieve that. We write this instead:

1
2
3
4
5
6
7

class PriceField(FlexField):
    clean = lambda v: int(re.sub('\s+', '', v))
    rules = [
        [lambda t: re_test(r'^[\s\d]*$', t), 'Enter a number'],
        [lambda v: v < 100000000, 'Specify adequate price'],
        [lambda v: v >= 1000, 'Specify price in rubles, ...'],
    ]

This works both in browser and server-side so we don’t need to duplicate logic or messages. Also having lots of similar fields we can reuse everything: fields, predicates, cleanup functions, rules, which are just lambdas plus text messages. So it really looks like:

1
2
3
4
5
6
7

class PriceField(FlexField):
    clean = clean.posint
    rules = [
        rule.posint,
        [test.lt(100000000), 'Specify adequate price'],
        [test.ge(1000), 'Specify price in rubles, ...'],
    ]

We can also employ inheritance:

1
2

class PriceField(PosIntegerField):
    rules = ...

Computed Properties

We really got more from this system than just validation. And first thing to show is computed properties. These are visibility, labels, choices and more. I’ll demo it with another tiny screencast:

Here we can see several aspects — when user selects “exchange to a car” a new drop-down appears, and when a label changes depending on who pays extra. Here is how it looks in code:

1
2
3
4
5
6

class ExtraPaymentField(FlexField):
    # Relabel depending on who pays extra
    label = lambda self: ('Max' if self.exchange_terms = MY_EXTRA else 'Min') \
                         + ' extra payment'
    # Show if there is an extra payment involved
    visible = lambda self: self.exchange_terms != NO_EXTRA

Additional benefit we get here is that we can compute visibility server-side and do 2 things: first, render a form in an appropriate state from the start (less flickering in the browser) and, second, discard any input we got from invisible fields automatically. Another example of additional benefit is calculated choices, able to get them on back-end we can check if value is one of them.

Other thing to note is that now we have dependencies, e.g. label text and visibility depends on exchange_terms field value. We could have added some tricky introspection, but we instead required programmers to specify dependencies explicitly:

1
2
3
4

class ExtraPaymentField(FlexField):
    @depends('exchange_terms')
    def visible(self):
        return self.exchange_terms != NO_EXTRA

This is in line with our meeting halfway principle and it worked this time too.

Passing Data

The last demo for today will show how easily we can pass data to client. Here I show how field values could be automatically selected:

To look through car models we need to pass their list to browser and we can do this simply by enclosing data structure with a function:

1
2
3
4

models = CarModel.objects.values_list('pk', 'title')

value = lambda self: first(pk for pk, title in models
                              if self.lookup in title)

This lambda uses models so they are translated to JavaScripts array of arrays and passed along with function. This is very handy when, for example, creating chained selects and under many other circumstances.

The Bigger System

You probably already noticed that there is a bigger system besides translator here. These are all its parts:

• declarative descriptions of fields, models, forms, filters, listings and detail pages,
• translation of clean, validate and property compute functions,
• transparent data forwarding.

There is the thing with tools that when you introduce some new very useful one you get a whole lot of new possibilities, you can make much faster something that took ages previously or you can even do things, which were impossible before.

This also works for developer tools. Introduction of this system had led to a population boom — we’ve got from 7 forms to 51, and you can double that number ‘cause we also autogenerate filtering form for each of them. We also got to 500+ database tables. All of this without loosing our sanity or growing our code base to some insane LOC number.

Decency

In python world we have this stigma on metaprogramming. Like it’s dirty and should not be used in any serious setting. It’s true that this is a complex technique that brings its costs. But as any other consideration this has its limits.

By promoting reasonable considerations to emotional and even kind of religious level we loose an ability to judge it mindfully. We dismiss something even before considering it seriously. And doing so we miss great solutions.

This worked for us. What will do for you?

Python gives us deep introspecting and metaprogramming capabilities including bytecode and AST support in the standard library. This facilitates unobvious ways to write code that writes code or otherwise alters its behavior. I will show several different examples of how it could be used for fun and profit.

Warming up

Look at this code:

1
2

>>> map(_ + 1, [1, 2, 3])
[2, 3, 4]

It maps a list through _ + 1 and gets a new list with each element incremented. This makes sense mnemonically, but can this code actually work in python?

And sure it can. In fact this is rather simple trick with operator overloading:

1
2
3
4
5

class Whatever(object):
    def __add__(self, y):
        return lambda x: x + y

_ = Whatever()

Here we create a class that produces lambda upon addition and then creates its instance named _. Just using common interface in an unusual way. We can even go all loose and make something like this to work:

1
2
3
4
5

# Sort guys by height descending
sort(guys, key=-_.height)

# Reduce my multiplication
factorial = lambda n: reduce(_ * _, range(2, n+1))

Returning lambda, obviously, won’t work anymore. However, we can return a callable object, which also returns callables as operations results. We are also intercepting attribute access here, shouldn’t surprise you after everything you’ve seen.

Honestly, I was too shy to use this in production. And you usually need only one or two lambdas in a file anyway, too few to bring an import, a dependency and all the complications. But suppose you are writing tests for functional support library, then you’ll need to create lots of small functions to pass them to your ones. A library like this would be useful in that setting.

Anyway, if you still think this idea is too weird then you should know that there are at least 3 libraries implementing it:

• whatever
• placeholder
• fn.py

But let’s move to something more interesting.

Pattern Matching

Now look at this and think if it is also feasible in python. This is supposed to recursively calculate a product of all the list elements:

1
2
3

def product():
    if []: 1
    if [x] + xs: x * product(xs)

If you familiar with pattern matching then the code makes sense:

• product is 1 for empty list,
• product is (first element) times (product of the rest of the list) otherwise.

On the other hand, python doesn’t work this way. x and xs are never defined and product() has no arguments. And yes, this really doesn’t work. This, however, does:

1
2
3
4
5
6

from patterns import patterns

@patterns
def product():
    if []: 1
    if [x] + xs: x * product(xs)

@patterns here rewrites function into something we meant. It can’t obviously be implemented as ordinary decorator, since calling original function will crash. So @patterns reads function code instead, parses it into abstract syntax tree, transforms that to a meaningful one and then compiles back:

Code → AST → New AST → New Code

And after transformation we get something like:

1
2
3
4
5
6

def product(value):
    if value == []:
        return 1
    if isinstance(value, list) and len(value) >= 1:
        x, xs = value[0], value[1:]
        return x * product(xs)

Which looks like normal python code, it is also much more verbose. That’s why we got into this whole pattern matching thing in the first place.

And again it was too awkward to use all this magic in production code, but I used code inspired by that:

1
2
3
4
5
6
7
8
9

# Use simple pattern matching to construct form field widget
TYPE_TO_WIDGET = (
    [lambda f: f.choices,           lambda f: Select(choices=f.choices)],
    [lambda f: f.type == 'int',     lambda f: TextInput(coerce=int)],
    [lambda f: f.type == 'string',  lambda f: TextInput()],
    [lambda f: f.type == 'text',    lambda f: Textarea()],
    [lambda f: f.type == 'boolean', lambda f: Checkbox(f.label)],
)
return first(do(field) for cond, do in TYPE_TO_WIDGET if cond(field))

This actually could be useful for many things: recursive functions, chatbots, binary protocol implementations and other occasions when you resolve to long if-elses or switch-cases. The general idea, however, comes beyond pattern matching.

AST Applications

This technique — abstract syntax tree transformation — is much more broadly useful. And the main reason is that trees capture language structure. Another trees virtue is that they are data structures, which are much more hackable than code strings.

Anyway, these are some things they facilitate:

Language extensions. With pattern matching being only one of them. Other examples are optional (or not) static typing, macros, … It is only limited by your imagination and tree transformation skills. You obviously need to stay within syntax though, unless you go for separate parser and AST, but this is a separate topic.

Optimizations. This is less obvious but we can inline calls, precalculate constant expressions, even make tail call optimization.

Code analysis. It’s much easier to analyze tree than just code string ‘cause it captures some of semantics of a language. We can implement some linters or editor plugins with the help of AST. Generally you won’t need to transform a tree for analysis, but imagine some linter suggesting code changes specific to your particular fragment not general advice. That would be cool.

Translation. To JavaScript to run things in browser, to SQL to automatically generate queries or stored procedures, to Python 3 after all.

These are various ways it flows:

Code → AST → New AST → New Code
Code → AST → JavaScript
Bytecode → AST → SQL

Looks like we need to dig a bit more into AST.

AST

First we need a way to get a tree. The easiest way is to first get source and then parse it. Two standard modules will help us with that:

1
2
3
4

import inspect, ast

source = inspect.getsource(func)
tree = ast.parse(source)

Sometimes you can’t get function source, most common example — inspect.getsource() doesn’t work with lambdas. Then you can still get to AST by decompiling bytecode:

1
2
3

from meta.decompiler import decompile_func

tree = decompile_func(lambda x: x + 1)

We are using third party module meta here, and despite saying that we decompile function it really does inspect and decompile its bytecode. Anyway, we’ve got our AST, let’s look at it:

This is quite a hairy tree for something as simple as lambda x: x + 1. But if we look at it a bit more closer we can recognize all the corresponding parts. Left branch is arguments, which have single argument named x, but no kwarg, vararg or defaults. Right branch is a return node returning a result of binary operation with an op of addition and operands of a name x and a number 1.

That makes sense, this is not however what we see as a result in python REPL:

1
2

>>> tree
<_ast.Lambda at 0x7f7e2359e490>

Yeah, it’s just a Lambda object. We can look up its attributes to get other nodes:

1
2

>>> tree.body
<_ast.Return at 0x7f3e75f869d0>

And this is what python AST actually is: a collection of objects stored in the attributes of each other. In addition I need to say that this objects don’t really have behaviour, they are just data. You can find full list of available nodes and their attributes in ast module documentation.

Inspecting AST this way could be too tiresome, we need some way to get a bigger picture. The simplest thing we have is ast.dump() utility:

1
2
3
4

>>> print ast.dump(tree)
Lambda(args=arguments(args=[Name(id='x', ctx=Param())], vararg=None, kwarg=None,
defaults=[]), body=Return(value=BinOp(left=Name(id='x', ctx=Load()), op=Add(),
right=Num(n=1))))

This is the same thing we saw in the picture, it is also useless for any non-trivial case. One more third-party library, astor, could help us out:

1
2
3
4

>>> print astor.dump(tree)
Lambda(
    args=arguments(args=[Name(id='x')], vararg=None, kwarg=None, defaults=[]),
    body=Return(value=BinOp(left=Name(id='x'), op=Add, right=Num(n=1))))

The same plus indents. It is already useful when you need to look at the AST closer, but for quick print debugging python code would be even better:

1
2

>>> print astor.to_source(tree)
(lambda x: (x + 1))

Parentheses are added everywhere to avoid dealing with operator precedence, but otherwise it’s the same code we parsed.

And the last thing we usually do to AST — compile it back and execute:

1
2
3

code = compile(tree, '', 'exec')
context = {}
exec(code, globals(), context)

Functions used here, compile() and exec(), are built-ins. And code is the same object type you can find introspecting func.__code__. By executing in a context we set all the names defined into that dict:

1
2

>>> context
{'func': <function __main__.func>}

So if all these were happening in a decorator we can get that function out of the context and substitute original one. Only the meat is left — walking the tree.

AST Traversal

Built-in ast module continues to aid us in this journey. It provides several utilities to ease tree walking and transformation. The lower level ones let us introspect nodes contents and iterate by them:

1
2
3
4
5
6
7
8
9
10
11
12

# Iterate fields with values
>>> ast.iter_fields(tree)
[('args', <_ast.arguments at ...>),
 ('body', <_ast.Return at ...>)]

# Iterate child nodes
>>> ast.iter_child_nodes(tree)
[<_ast.arguments ...>, <_ast.Return ...>]

# Just list available fields
>>> tree._fields
('args', 'body')

First of higher level ones is NodeVisitor — a base class implementing visitor pattern, the idea is to subclass it and define methods like visit_<NodeClass>():

1
2
3
4
5
6

class NumberFinder(ast.NodeVisitor):
    def visit_Num(self, node):
        print "Found number literal", node.n

def find_numbers(tree):
    NumberFinder().visit(tree)

This simple example finds all the number literals in given code. Just an example, but it could be grown into some magic number finding tool and wrapped into linter or code editor plugin.

The second base class ast provides is NodeTransformer. It works almost the same, but you can change a tree by returning new nodes from visit methods:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

class ExprOptimizer(ast.NodeTransformer):
    def visit_BinOp(self, node):
        # Call .generic_visit() to transform sub-nodes
        node = self.generic_visit(node)
        # If both operands are just numbers then precalculate
        if type(node.left) is Num and type(node.right) is Num:
            op = AST_TO_OP[node.op.__class__]
            return Num(n=op(node.left.n, node.right.n))
        else:
            return node

AST_TO_OP = {
    Add: operator.add,
    ...
}

This one precalculates any expressions on number literals, quite neat for just a bunch of lines.

And last but not least we can use old and good recursive tree walking. This snippet makes python function return last expression in a way Ruby, CoffeScript and Clojure do:

1
2
3
4
5
6
7

def return_last(tree):
    if not hasattr(tree, 'body'):
        return
    elif isinstance(tree.body[-1], Expr):
        tree.body[-1] = Return(value=tree.body[-1])
    else:
        return_last(tree.body[-1])

Note this code is simplified and hence broken, e.g. it will insert return into the end of while loop if there is an expression there.

Going Forward

This was an overview of what you can do with AST, and there were lots of things. This is, however, only a preface to the actual story. Next time I will concentrate on translation and will describe it in detail capturing a real use case.

P.S. Second part is available here.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

<span id="colored-counter">0</span>
<input id="color"></input>
<button id="inc"></button>

<script>
$('#color').on('keyup', function () {
    $('#colored-counter').css('color', this.value);
})

$('#inc').on('click', function () {
    var oldValue = $('#colored-counter').html();
    var newValue = 1 + Number(oldValue);
    $('#colored-counter').html(newValue);
})
</script>

Could be written as:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

<span id="colored-counter">0</span>
<input id="color"></input>
<button id="inc"></button>

<script>
var state = {color: '', value: 0};

function updateUI() {
    $('#colored-counter').css('color', state.color);
    $('#colored-counter').html(state.value);
}

$('#color').on('keyup', function () {
    state.color = this.value;
    updateUI();
})

$('#inc').on('click', function () {
    state.value++;
    updateUI();
})
</script>

This demonstrates React idea in its simplest (and a bit bastardized way). We get to this later, now we should answer a reasonable question: how is this better? Let me draw you a picture. This is how first variant works:

And this is how second one does:

Doesn’t look that much better, but let us think scale. Over time our UI will get more events and elements, some events will update more than one element, some elements will be updated by more than one event. So we will get this:

And even with our bastardized React-like code we get:

In first picture we have N events, M elements and up to O(N*M) links. In second one we have just N event links and M element links. Overall we got from potential O(N*M) complexity down to O(N+M), not bad for a simple trick.

Now it looks like we got something. But what about all those magical powers? Ah, we almost got them too.

Magical Powers

Basically all magical powers mentioned come from an ability to write:

1
2
3
4
5
6

// save
var cannedState = deepcopy(state);

// load
state = cannedState;
updateUI();

This is facilitated by 2 facts:

• we have a single explicit state,
• we can update UI to comply to arbitrary state.

That’s it. We can write serialized canned state to local storage, send it to server, manage a history of canned states. And this will give us page reload protection, persistence and undo/time-travel respectively.

Confused? Let’s write some code. That’s how we protect from accidental page reload:

1
2
3
4
5
6
7
8
9
10
11

function updateUI() {
    // Save latest state to local storage
    LocalStorage.set('state', JSON.stringify(state));
    // ... continue as usual
}

// Load saved state from local storage on page load
$(function () {
    state = JSON.parse(LocalStorage.get('state'));
    updateUI();
});

And that’s how we get time-travel debugging:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

<span id="time-pos"></span>
<button id="back">Back</button>
<button id="next">Next</button>

<script>
var time = {history: [], pos: 0};

function updateTimeUI() {
    $('#time-pos').html('Position ' + time.pos + ' of ' + time.history.length);
}

function saveState() {
    time.history.push(deepcopy(state));
    time.pos++;
    updateTimeUI();
}

$('#back').on('click', function () {
    // Move history pointer
    time.pos--;
    updateTimeUI();
    // Load historic state
    state = deepcopy(time.history[time.pos]);
    updateUI();
})
$('#next').on('click', function () {
    time.pos++;
    // ... same
})

// ...

function updateUI() {
    // Save state to history on every change
    saveState();
    // ... continue as usual
}
</script>

On each change of state we push its deep copy to history list and later we can restore that state by simply copying it from history and updating UI to fit it. I created a working clickable demo to illustrate it even more.

Note also how we use the same pattern here? time is the state of time-travel sub-application and updateTimeUI() is its update function.

Now we can build from that. Undo is just time-travel for user not developer, by saving history to local storage we can preserve it on page reload, by catching errors and sending them along with serialized history to our bug tracker we can automatically reproduce errors our users face.

I should note that React doesn’t have all these magical powers, at least out of the box, since it separates app state into pieces and hide it in components.

Pure Function

Why do we need pure functions, immutable data and virtual DOM? These are optimizations and to some extent simplifications, not core to idea. Still let’s us follow the way that led to them.

First we need a slightly more intricate example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

<span id="count">2</span>
<ul>
    <li>hi</li>
    <li>there</li>
</ul>
<button id="add"></button>

<script>
var state = {items: ['hi', 'there']}

function updateUI() {
    $('#count').html(state.items.length);
    // Compare ul.childNodes to state.items and make updates
    // ...
}

$('ul').on('click', 'li', function () {
    state.items.splice($(this).index(), 1);
    updateUI();
})

$('#add').on('click', function () {
    state.items.push(getNextString());
    updateUI();
})
</script>

Several things to see here:

• there is a duplication between prerendered html and initial state,
• update became complex ‘cause we need to compare data structure to DOM.

There is a simpler way to do this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

<div id="ui"></div>
...

<script>
...

function render(state) {
    var span = '<span id="count">' + state.items.length + '</span>';
    var lis = state.items.map(function (item) {
        return '<li>' + item + '</li>';
    });
    return span + '<ul>' + lis.join('') + '</ul>'
}

function updateUI() {
    $('#ui').html(render(state));
}

...
</script>

Here render() is just pure function from app state to html. And we want it to be pure so that state will be the single thing defining UI.

In a way UI was a function of state even in our bastardized code, it just wasn’t clearly obvious ‘cause we updated UI, not calculated and rebuilt it from scratch each time. There also was a possibility to screw up and write updateUI() so that UI state is defined both by its own previous state and application state, thus using render is not really an optimization, but a simplification and a guard.

But anyway that’s it on pure functions.

Virtual DOM

Now look at those render() / $().html() pair, they build entire representation from scratch on each event, probably on every key press. This sounds slow, so we use another optimization — virtual DOM:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

var root = document.getElementById('ui');
var prevState = state, prevTree = [];

function render(state) {
    // Virtual DOM is really just a tree of JavaScript objects or arrays
    return [
        ['span', {id: 'count'}, state.items.length],
        ['ul', {}, state.items.map(function (item) {
            return  ['li', {}, item]
        })]
    ]
}

function updateUI() {
    var vTree = render(state);
    var diff = vDiff(prevTree, vTree); // Just a diff on data structures, haha :)
    vApply(root, diff)                 // Apply series of patches to real DOM

    prevState = deepcopy(state);
    prevTree = vTree;
}

If any of this diffing/patching sounds complicated to you then you shouldn’t worry React or standalone virtual DOM implementation got you covered. But that’s really not that complicated. If you know how to write diffing algorithm then you can surely implement this yourself.

Note that what we just did is a premature optimization. Current simple example will do just fine with naive render-to-string implementation above. Surprisingly, most of SPAs out there will do just fine as well, browsers are hell fast these days. I want to stress this once more – for an average app you can skip React or other virtual DOM at start and only go for it once it gets too slow (or never).

Another stone in Reacts direction: it’s virtual DOM is one of the slowest ones across the block.

Immutability

Noticed all that deepcopy() calls lying around? This looks like a waste (it really is cheap, but bear with me). The idea behind immutable data structures is to build new state based on previous one instead of copying and then mutating it.

Maybe it is still not clear enough, so I’ll show you. That’s how we make immutable objects in JS:

1
2
3
4
5
6
7
8
9
10

var object = {
    a: {x: 1, y: 2},
    b: {text: 'hi'}
}

// Now instead of object.a.y = 3 we do
var object2 = {
    a: {x: object.a.x, y: 3},
    b: object.b
}

Look how we reuse object.a.x and object.b. This not only saves us from copying, but effectively makes diffs way faster: before diffing object.b and object2.b we just check if they are the same object, which is referential equality, and done — diff is empty, no need to go all the way down.

One thing to add is this naive approach to immutable collections is not only cumbersome, but also ineffective. Suppose you have an object with lots of keys and a value of one of them changes, you now need to create a new object with all the same keys (you can still reuse values). You probably don’t use such objects, but what about array? Let’s see how that looks like:

1
2

var prev = ['a', 'b', 'c', ..., 'z'];
var next = prev.slice(0, n-1).concat([newValue]).concat(prev.slice(n));

So we need to copy the whole thing. Here is a better way to implement immutable sequences:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

var prev = {
    '0-3': {
        '0-1': {0: 'a', 1: 'b'},
        '2-3': {...},
    },
    '4-7': {...}
}
var next = {
    '0-3': {
        '0-1': prev['0-3']['0-1'],
        '2-3': {
            2: 'hey',
            3: prev['0-3']['2-3'][3]
        }
    },
    '4-7': prev['4-7']
}

We can see that we only create log N new objects here and the rest is reused, so even less copying and faster diffs. You also don’t need to really deal with all this trees. There are great ready to use implementations out there. E.g. this is how our first example looks with a help of Immutable.js:

1
2
3
4
5
6

var object = Immutable.fromJS({
    a: {x: 1, y: 2},
    b: {text: 'hi'}
})

var object2 = object.setIn(['a', 'y'], 3); // object remains intact

It gives you nice API, efficiency and protection against accidentally writing to supposed to be immutable collection. Also take a look at mori — a set of immutable collections extracted from ClojureScript.

Note that React actually uses naive immutable by convention collections.

Wrap-up

It may look from my notes that React is not a good tool, but it is actually very useful one. It may not have magical powers and top speed, but it got components and is still pretty fast. It also offers ecosystem and community.

And there is one more way to look at it. React made a significant push to front-end development in an interesting direction, including all those virtual DOMs and immutable collection libraries.

Anyway, now that you know all this, you can judge better on what to use and how to structure your code.

So what should conscious designer do?

Possible strategies

Most languages don’t have any explicit strategy for backward incompatible changes, which is pitiful. Mainly because not having a strategy means you are using the worst one. But let’s review possible strategies.

Grow as it goes, then break big

This results into separation like Python 2/3, Perl 5/6 or even in total abandonment. The major issue is that it creates a gap that most people don’t have enough incentive to jump over. It creates additional headache to library developers making hard to support their creations across incompatibility boundary. This in its turn causes chicken and egg problem of no libraries, no users. In its worst separation could divide both ecosystem and community in half, lessening the project considerably.

Move fast, break things

Works well in earlier development stages. I never complained when upgrading Node.js from 0.1.x to 0.2.x broke my app as well as several dependencies. However, this effectively reduces library availability, since many authors just can’t or interested enough to keep up with the development pace. I was astonished to hear from some of my ruby friends that they avoid using libraries not updated within 3 months as possibly abandoned ones.

It also poses a strain on language users forced to change their coding habits and to constantly seek for and adapt a replacement libraries for abandoned ones. Some of them consequently settle with particular language or framework version causing an erosion of a community and ecosystem.

Stay small

… and useless. A viable strategy for very domain specific or otherwise having narrow field languages, e.g. for education. In any other case wider audience will need language to be extended to be useful for them. The alternative is to stay small …

… and provide language extending capabilities. Seems like a good one. However, it hadn’t quite worked for lots of lisps and forths. The common issue is that by not providing enough common ground you encourage separating into dialects.

Language of each project gets tailored to project needs, to particular team style. Communities divide, codebases become incompatible effectively narrowing the language for its users and causing confusion for newcomers.

The other issue with both flavors of this approach is that problem is not solved, it’s only reduced. There is still no strategy to evolve the core.

Rolling deprecation

This is when you introduce new things leaving old intact, but marking them as deprecated. After several releases deprecated things are removed and new things take their place. Using this strategy you can continuously introduce changes while still providing a compatibility guarantee for several releases. There are, however, some considerations to this strategy.

First, you’ll need new API each time you update something in backwards incompatible manner. Say you have a function to parse URLs and it returns a dict of query parameters. It however doesn’t handle repetitive params well, it just captures last value. So you decided to update it to return a list instead. This change could break someone’s code, so you end up with one of:

1
2
3
4
5

def parse(url, capture_all=False):
    ...

def parse_all(url):
    ...

Your users, respecting deprecation, litter their code with ugly parse(url, True) or parse_all(url) calls. And when finally you get rid of an old parse you need another round of deprecation to return to sane API. On function level this leads to do_something_ex() things, on module level we can see urllib2, newforms and better-assert.

Second, you’ll obviously need to ship several implementations for things you change. This hardens managing whatever you are doing a bit. But making life easier for broader group, whatever users, by making it harder for narrower one, whatever developers, is generally a good trade-off.

Third, gradual change is not always possible. Say you want to change how the / operator works in your language. To be compatible it should work the old way in old code and a new way in new one.

Rolling a language

Making an operator work differently depending on context seems impossible, but also offers a hint to resolve both itself and API uglifying issue. So obviously first we need a way to tell new code from old. The simplest thing is just stating this explicitly, and this is already used in Python 2:

1

from __future__ import division

and in Perl 5:

1

use feature 'say';

Python version has file scope and Perls lexical one. Perl also goes farther by bundling features into language versions. E.g. by:

1

use v5.16;

you can turn on all 5.16 features, this can also deprecate some old features or alter their behavior. Any old code without that use statement or with the one with earlier version continues to behave the exact way it did. So we can change a language in a backward incompatible manner while still providing compatibility with older code.

So this is it, the idea I started this post to promote. Lower I’ll just address any issues/objections that I can think of.

Issues

Let’s start from the ones from rolling deprecation section. First (ugly APIs) and last (no way to break) are resolved automatically, second – the need to ship several implementations bundled – remains. We can manage that by carefully storing all active implementations and providing a limit on supported versions, e.g. make use v5.10 an error in 5.20 version of a language. We’ll need to make this span large and stable enough though.

There is also an issue of how do we start this process. It’s however rather trivial, say our language is at version 3.4 now, when to add our rolling model in 3.5, we add some form of use statement there and in absence of it just assume 3.4 semantics.

Easy parts

These ones are almost automatically solved by version declaration, still some elaboration will probably close unneeded questions.

Syntax changes. We have a chicken and egg problem like use is part of syntax and it can alter syntax itself. This is however only a problem if we follow Perl 5 lead and allow it anywhere. Although it’s cool, it has no relation to solving real problem of “using this library written for Blab 2.7 in my Blab 3.4 code”. So we can just require every file start with use statement and then parse the rest with appropriate parser. Yes, we’ll need to ship several parser implementations, they could share code though.

Semantic changes. Like we have the same syntactic construction or operator and it should behave in a new way. The beauty of declared version and separate parsers is that we can substitute a thing during parsing. For example, if we get tired of Python semantics of:

1
2
3
4

try:
    ...
except E1, E2: # really means "except E1 as E2:",
    ...        # not catch both as you might expect

We can just make it to parse into the same thing as except (E1, E2). Altering an AST node resulting from parsing an operator is even easier.

Built-in changes. Having all new and old built-ins implementations we’ll need only arrange that globals are updated on use statement. Globals should be scoped lexically, at least at file level, for this to work. Note that by altering globals we handle all of addition, removal, rename and change of behavior of any built-in.

Standard library changes. The easiest way is just shipping several versions of it and patching import statement to load appropriate library version. Different versions of standard libs can share code for efficiency, but that’s optional.

Passing over boundary

Real issues start to arise when we pass something from newer code to older one or vice a versa. Say we have an instance of built-in or standard library type which has changed and pass it to the code that expects to treat it as an instance of an older version of same type.

The simplest thing to do is to provide converters and require a new code interfacing with an older one to use them. This will work, however, it will place significant burden on every forward-looking user of that old code. This is also not future compatible in a sense that if suddenly that old code is updated everyone needs to remove converter calls. And even if we manage somehow to automatically intercept all calls over boundary and convert everything there still be an issues like it could be slow or it would be impossible to share data by reference between newer and older code.

Given all these obstacles, being able to gradually update your code even with a use of converters is significant improvement over major break with just no way to call older code without first updating it. And converters are quite simple thing to implement. But let’s take a view into some more elaborate ways to overcome type changes.

Say if a new version of a type has same internal representation and only a new interface then we can pass it as is and rely on some lexical dependent method substitution. This could sound like some unscientific magic, but there is a corresponding feature in ruby 2.0 and sure there is a perl module for that. We can even incorporate active language version into method call semantics, this way changed types should just respond to all their historical interfaces in a support window, which probably won’t be more than two.

What if internal representation of a type changes? The thing is that shouldn’t matter if nobody looks inside. We should still support all the interfaces and we’ll be good. This puts a restriction on a kind of access language users, including library authors, have to instances of built-in types. E.g. if instance data is represented with C struct then we shouldn’t allow direct access to its members, casting, etc. Everything should be done via functions or at least macros, which could be updated to handle newer representation.

Wrap-up

There are obviously some things I haven’t addressed. And this approach doesn’t make language developers lives easier. It could however make it more fun both for developers and their users. Cause this way things don’t need to stay broken or weird cause that’s how they are. An ability to change language in a backward incompatible manner should in the end bring better languages for everyone. It also feels like freedom.

Do you know what underscore is? In its most general it’s a JavaScript library that makes life better. For whoever writes in JavaScript. I mostly write Python and I also wanted my life better, so I went ahead and wrote a similar library for Python. But the thing is you need to understand a magic behind something to properly replicate it. So what’s the magic behind _?

The problem

To answer this question we should look at problems this kind of libraries solve. To get some ground I’ll illustrate them with code. That will be in python, but all the ideas are universal, so don’t be afraid.

A piece of entangled code

This messy piece of code was taken from a real project and slightly simplified:

1
2
3
4
5
6
7
8
9

images = []
for url in urls:
    for attempt in range(DOWNLOAD_TRIES):
        try:
            images.append(download_image(url))
            break
        except HttpError:
            if attempt + 1 == DOWNLOAD_TRIES:
                raise

There are several things entangled in here, but my point is that this could be written much shorter:

1
2

http_retry = retry(DOWNLOAD_TRIES, HttpError)
images = map(http_retry(download_image), urls)

If it seems hard at first, then it’s okay. It involves some functions and control flow abstractions most probably new to you. Once you get used to it you’ll find that the latter variant is not only shorter, but is also simpler.

Dirty dictionary

But let’s go on and clean some dirty dictionary:

1
2
3
4
5
6

d = {}
for k, v in request.items():
    try:
        d[k] = int(v)
    except (TypeError, ValueError):
        d[k] = None

Here we go through dictionary and clean its values by coercing them to int. Or to None if that is impossible. Cleaning input and ignoring malformed data is quite frequent task and yet it takes so much effort. This is the way I want to write that:

1

walk_values(silent(int), request)

And it’s entirely possible with funcy. But let’s move to the next one.

Pairwise iteration

This code checks if a sequence is ascending:

1
2
3
4
5
6
7
8

prev = None
for x in seq:
    if prev is not None and x <= prev:
        is_ascending = False
        break
    prev = x
else:
    is_ascending = True

Ah, iterating over a sequence and keeping track of a previous element. How many times had you done that? There should be a function to abstract it:

1

is_ascending = all(l < r for l, r in pairwise(seq))

And pairwise does exactly that. It enables us to iterate by sequence adjacent pairs. So we just need to check that all of them are ordered accordingly.

…

All these examples have one common property — red variants have more code. And more code:

• takes longer to write,
• takes longer to read,
• takes longer to debug,
• contains more bugs.

Obviously, underscore, funcy and friends help us write less code (at least in these three examples). But how do they do that?

Extracting abstractions

Let’s take another look at the first example. It does three things in a single blob of code:

1
2
3
4
5
6
7
8
9

images = []
for url in urls:
    for attempt in range(DOWNLOAD_TRIES):
        try:
            images.append(download_image(url))
            break
        except HttpError:
            if attempt + 1 == DOWNLOAD_TRIES:
                raise

I highlighted every aspect of this code with separate color:

• image download (green),
• retries on fails (red),
• iteration through urls and result collection (blue).

As you can see, three colors are interleaved here. This hints that corresponding aspects are entangled. And by “entangled” I mean they can not be reused separately. Say we need retries on fails in some other place, we will probably end up copying the whole block and updating it somewhat. Not exactly the best practice “code reuse”.

If, on the other hand, we managed to separate reties then our code will look like:

1
2
3
4
5
6
7

def retry(...):
    ...

http_retry = retry(DOWNLOAD_TRIES, HttpError)
images = []
for url in urls:
    images.append(http_retry(download_image)(url))

Now red code is nicely grouped at the top. Green and blue are still mixed, but now they represent a pattern so common that most modern languages have a builtin function to handle that:

1
2
3
4
5

def retry(...):
    ...

http_retry = retry(DOWNLOAD_TRIES, HttpError)
images = map(http_retry(download_image), urls)

This last variant has some lovely traits: each part of a task at hand (downloading images) appear only once, the whole iteration aspect is handled with a single map() call and retries are abstracted out into the retry function.

Extracting common behavior into a higher order functions is a first trick underscore and funcy use to make your life better.

Hiding low level

It’s time to go back to second example. I’ll throw away error handling to make snippets more even:

1
2
3
4
5

# Using function
walk_values(int, request)

# Using dict comprehension
{k: int(v) for k, v in request.items()}

Now they are both one-liners, so how is first one better? Let’s identify every single distinct component of each code variant:

1
2
3
4
5

# 3 components
walk_values(int, request)

# 8 or so components
{k: int(v) for k, v in request.items()}

The second one looks like rainbow. But besides looking nice this means each time you write or read it you need to load all those components into your head, taking up all your cognitive resources. This is how first line is better.

That could be highlighted in even more obvious manner:

1
2
3
4

walk_values(int, request)

# red are low-level details
{k: int(v) for k, v in request.items()}

This way we can see that about a half of the second line is low-level details. And low-level mean you don’t need to know all those details to understand what’s going on.

Hiding low-level details is the second way such libraries make your life better.

Enriching our language

I’ll translate the last example into natural language:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

prev = None
for x in seq:
    if prev is not None and x <= prev:
        is_ascending = False
        break
    prev = x
else:
    is_ascending = True
# Starting with empty prev, iterate over seq, bookkeeping prev actuality,
# on each cycle if prev is present and current element is less or equal than it
# then set is_ascending to False and break.
# If loop wasn't broken set is_ascending to True

is_ascending = all(l < r for l, r in pairwise(seq))
# set is_ascending to all left elements being smaller than right
# in adjacent pairs of seq

Obviously, more code emits more text. Higher level code generates an explanation utilizing higher level abstractions. This way we can use bigger building blocks not only in coding, but in problem solving.

And this is the third way _ makes your life better.

Wrap-up

All the things we came through are completely language independent. So there gotta be underscore for every language? Not quite, and more importantly a straight-forward port is not always a great idea: common behaviors to abstract vary per language and especially per application. The right approach would be to follow core ideas. Or look around if someone have already done that.

Here are some leads for you to take:

P.S. You may also want to look at Scala if you are using JVM and at F# if it’s .NET.

P.P.S. Please stop commenting on Hacker News, a controversial penalty is killing this post. Use reddit thread instead. Sadly, HN is not a place for discussions anymore.

I will move from basic ideas building upon them as needed and diving into more and more specifics as the post goes.

Completeness and redundancy

Let’s start from some general considerations. I define abovesaid completeness as a characteristic of invalidation procedure describing how frequent and under what circumstances data can become dirty and how long it will remain accessible. And redundancy will be a frequency and a volume of cache invalidated needlessly.

An example will help us perceive these two concepts better. Let’s look at common time-invalidated cache. On one hand, it inevitably leads to dirty data between update and cache timeout, making this algorithm inherently incomplete. On other hand, we can easily reduce incompleteness by reducing cache timeout, which, in it’s turn, will increase redundancy – clean cache data will be invalidated more frequently, which will lead to more cache misses. And for ideal completeness (no dirty data) we need to set timeout to zero.

There are lots of scenarios where it’s ok to use stale data: popular articles list doesn’t change fast and it’s not a big deal if user count of your social network is off by a couple of thousands. But then there are some scenarios where you need immediate invalidation, go to next section for that.

Event-driven invalidation

Probably, the only way to achieve ideal invalidation completeness is to invalidate each time you change data. These are elements of such system we need to think about:

• cached things,
• events,
• a dependency matrix explaining what thing to invalidate on what event.

There are obviously different strategies to define those in your code. I’ll start from simplest one – manual definition.

Coding dependencies by hand

First, cached things, they will probably look like this (in sort of python pseudo-code):

1
2
3
4
5
6
7
8
9

# An ellipsis means some code or instruction to get corresponding data
register_cached_item('post_by_id', ...)
register_cached_item('posts_by_category', ...)
register_cached_item('recent_posts', ...)
register_cached_item('posts_by_tag', ...)

# Used like
post = get('post_by_id', 10)
invalidate('post_by_id', 10)

Remember this is pseudo-code. register_cached_item() doesn’t need to be a function call, it could be a decorator in Python, a macro in lisp or a class in Java.

Ok, say we have some place in our code that adds post, we then need to add something like this there:

1
2
3
4
5
6
7
8
9
10
11

# ...
# Just added post, need to invalidate things
invalidate('post_by_id', post.id)
invalidate('posts_by_category', post.category_id)

# It could be recent
invalidate('recent_posts')

# Invalidate for all tags
for tag in post.tags:
    invalidate('posts_by_tag', tag)

And if we delete post somewhere we need to make invalidation there too. We should obviously abstract our code into some invalidate_post() function and call it from both places. So far, so good. What about updating? The thing is it’s not enough to just call our invalidation procedure from there:

1
2
3
4
5
6

post = get('post_by_id', 10)
assert post.category == 1
post.category = 2

update_post(post)
invalidate_post(post) # 'posts_by_category' not invalidated for category 1!

We need to invalidate on both old and new states of a post on update. How we get old state is a separate not an easy question, but suppose we have both states, should we just call invalidate_post() twice for each of them? Not so efficient, but that would work.

There is a problem with our code though. It’s tightly coupled – update logic knows about cache logic. Even bigger problem is that our cache logic is scattered. We define cached thing in one place and invalidate in another or several other places. This means there will come some day when someone will add a cached thing and just forget to add corresponding invalidation, producing hard to find bug.

Bringing things together

Fortunately, there is a single solution to both problems – events. We can write fetching and invalidation logic for each thing in one place and then register both cached thing and event listener:

1
2
3
4
5
6
7

def post_by_id(id):
    # ...

def invalidate_post_by_id(post): # event signature here
    invalidate('post_by_id', post.id)

register_cached_thing('post_by_id', fetch=post_by_id, invalidate=invalidate_post_by_id)

As invalidate procedures would be tiresomely repetitive we can dry this up to (and even further using particular language sugar features):

1
2
3

def post_by_id(id):
    # ...
register_cached_thing('post_by_id', fetch=post_by_id, arg=lambda p: p.id)

Looks like we’ve come up with pretty solid system. It’s dry, but retains flexibility of still mostly manual system. We, however, need to declare each thing we plan to cache in advance. We also need to provide argument constructing functions. There got to be a smarter way.

Automatic invalidation

We have not even started utilizing a power of ORM. Its query is not a mere text and plain arguments, it is a structure which includes condition tree. And some smart code could use that information to determine when query cache should be invalidated, saving work for lazy guys like me.

Suppose we cache a query:

1

select * from post where category_id = 2 and published

We should drop that cache when we add, update or delete post with its old or new state satisfying category_id = 2 and published condition. So the time we save that cache we should write along “invalidator” like that:

1

category_id = 2 and published: K1 -- K1 is above query cache key

Then if some post changes we look up all invalidators and check their conditions with post at hand, deleting cache keys corresponding to holding ones. That could become very inefficient once we’ll have lots of queries cached.

The reason we will end up with lots of invalidators is that we have cached lots of different queries. In common use, however, most queries will differ only in parameters not structure. Maybe separating those will help us? Let’s try on larger example. Here be the queries:

1
2
3
4
5
6
7
8

select * from post where category_id = 2 and published           -- K1
select * from post where category_id = 2 and published limit 20  -- K2
select * from post where category_id = 3 and published           -- K3
select * from post where category_id = 3 and not published       -- K4
select * from post where id > 7                                  -- K5
select * from post where category_id = 2 and published or id > 7 -- K6
select * from post where category_id in (2, 3) and published     -- K7
select * from post where category_id = 2 and id < 7              -- K8

Unseparated invalidators first:

1
2
3
4
5
6
7

category_id = 2 and published:           K1, K2
category_id = 3 and published:           K3
category_id = 3 and not published:       K4
id > 7:                                  K5
category_id = 2 and published or id > 7: K6
category_id in (2, 3) and published:     K7
category_id = 3 and id < 7:              K8

We can use some tricks to make this more regular. ored conditions could be split into two, in is basically syntax sugar for or and boolean tests could be substituted with equalities. Applying these we get to:

1
2
3
4
5

category_id = 2 and published = true:  K1, K2, K6, K7
category_id = 3 and published = true:  K3, K7
category_id = 3 and published = false: K4
id > 7:                                K5, K6
category_id = 3 and id < 7:            K8

Note that after this transformation all conditions became simple conjunctions. And we are finally ready to separate condition scheme from data:

1
2
3
4
5
6
7
8
9
10
11

-- Schemes
category_id = ? and published = ? -- S1
id > ?                            -- S2
category_id = ? and id < ?        -- S3

-- Conjunctions
S1:2,true:  K1, K2, K6, K7
S1:3,true:  K3, K7
S1:3,false: K4
S2:7:       K5, K6
S3:3,7:     K8

Time to try modeling invalidation procedure. Say we are adding this post to our stock:

1

{id: 42, category_id: 2, published: true, title: "...", content: "..."}

Looking at S1 we can see that there is at most one conjunction of that scheme satisfying our state! Even better, we can build it from scheme and object data by looking at field values: each known value is equal only to itself. Too bad the trick won’t work with S2 cause our post id, 42, is greater than many things. To find what queries of scheme S2 should be invalidated one needs to look through all S2:* conjunctions and that could be a lot.

This is probably a case for trade-off. Dropping all conditions but equalities we will sacrifice invalidation granularity, but simplify and speed up the procedure. Simplified invalidators will look like:

1
2
3
4
5
6
7
8
9
10
11

-- Schemes
category_id = ? and published = ? -- S1
                                  -- S2, an empty scheme
category_id = ?                   -- S3

-- Conjunctions
S1:2,true:  K1, K2, K6, K7
S1:3,true:  K3, K7
S1:3,false: K4
S2::        K5, K6 -- no data for S2
S3:3:       K8

There are some points worth noting here. First, S2 is now an empty scheme with a single empty conjunction which is always true, which means K5 and K6 will be invalidated on any post change. Second, schemes are now just sets of field names, pretty neat.

So far I tried to stay language and platform agnostic, but that journey came to an end. Welcome to dirty reality in the next section.

Implementation tips

• A best way to represent scheme is probably alphabetically sorted list of field names, it’s easily serializable and it makes building a conjunction for an object pretty straightforward.

• Extracting conjunctions from a query tree could be tricky. One might want to employ fuzzy logic: look at not (f > 0 and g != 1), if we drop f > 0 right away, we’ll end up with g = 1, which is not equivalent to f <= 0 or g = 1. Lost empty conjunction along the way!

• Considering invalidator structure (sets) and taking into account that it is generally a good idea to keep all interdependent data together (cache and invalidators) this is an excellent case to use Redis. Using it we can easily add keys to conjunctions, we can even do SUNION of conjunctions to fetch all dirty keys in one shot.

Sure, I went ahead and used that tips in my Django caching solution. Hope it or the ideas it embodies would be helpful.

Let’s find that out.

Current state

Currently in a decorator you need to create a wrapper, update it’s metadata and then return it. You also need to pass arguments and result value in and out carefully. A typical pattern would be:

1
2
3
4
5
6
7
8
9
10

from functools import wraps

def some_decorator(func):
    @wraps(func)
    def wrapper(*args, **kwargs):
        # ... do something before
        result = func(*args, **kwargs)
        # ... do something after
        return result
    return wrapper

And if you want decorator with arguments it becomes even harder:

1
2
3
4
5
6
7
8
9
10

def some_decorator(before, after):
    def decorator(func):
        @wraps(func)
        def wrapper(*args, **kwargs):
            before()
            result = func(*args, **kwargs)
            after()
            return result
        return wrapper
    return decorator

Look at that code for a minute. What a mess! And this is not about typing, it’s really hard to see through to the point of it.

Let’s see what is meaningful and what is a boilerplate here. What we care about is concentrated in wrapper function, it’s before and after calls, how they surround original function call, what arguments are passed to it and what is returned. The name some_decorator and its arguments also matter. Anything else – nested functions and returns, @wraps and all extra indent – is just code pattern waiting to be abstracted.

Removing a boilerplate

What we probably want to see (or write) is some flat syntax with code showing only wrapper semantics. Something like:

1
2
3
4
5

def some_decorator(...):
    # ... do something before
    result = func(*args, **kwargs)
    # ... do something after
    return result

Obviously func, args and kwargs should come from somewhere as well as all the magic turning plain function into decorator. Suppose we have @decorator to handle the conversion and the easiest way to provide all values is just enclosing them as some_decorator arguments:

1
2
3
4
5
6

@decorator
def some_decorator(func, args, kwargs):
    # ... do something before
    result = func(*args, **kwargs)
    # ... do something after
    return result

Calling decorated function with the same arguments as its wrapper is a pattern so common that we should abstract that too. Also, our decorator may become too cluttered once we start adding its own arguments. To keep this clean we can substitute three call related arguments with a single call object:

1
2
3
4
5
6

@decorator
def some_decorator(call):
    # ... do something before
    result = call()
    # ... do something after
    return result

Looks simple and to the point. Something we were looking for. And once we have a clear view we can go ahead and implement the magic. Actually, you can download it from pypi right now and import it with from funcy import decorator.

Advanced usage

We seemingly lost a couple of features in our quest for simplicity: access to call arguments, function itself, ability to call function differently and create decorators with arguments. But that’s not the case really, we can pack all these into call object. I’ll go through everything with better examples than you saw so far.

Accessing function arguments

You didn’t expect a post about decorators without logging example, did you? Anyway, it’s an excellent way to show how you can access not only call arguments but decorated function itself:

1
2
3
4
5

@decorator
def log(call):
    print "Calling %s with args %s and kwargs %s"         \
        % (call._func.__name__, call._args, call._kwargs)
    return call()

As you can see, everything is underscore-prefixed, this is done to avoid clashes with function argument names, the values of which are binded to call object as attributes. Pretty handy if you are writing something more specific than logging decorator. Look at this simplified login_required decorator to get a foretaste:

1
2
3
4
5
6

@decorator
def login_required(call):
    if call.request.user.is_authenticated():
        return call()
    else:
        return redirect(LOGIN_URL)

Altering calls

Is considered a bad practice cause it makes code harder to read. However, this could be useful occasionally and I am not into childproofing anyway. The obvious way to do it is just using plain function carefully stored in call._func:

1
2
3
4

@decorator
def int_args(call):
    """Coerces any function arguments to ints"""
    return call._func(*map(int, call._args))

But a common use-case of passing some extra data to function could be written with more elegance:

1
2
3
4
5
6
7
8

@decorator
def with_phone(call):
    phone = Phone.objects.get(number=call.request.GET['phone'])
    return call(phone) # phone is added to *args passed to decorated function

@with_phone
def some_view(request, phone):
    # ... some code using phone

This works with named arguments too, and is probably a better way since you won’t run into problems with arguments order.

Decorators with arguments

To get these you just add your arguments after call as in this control flow abstracting retry decorator:

1
2
3
4
5
6
7
8
9

@decorator
def retry(call, tries, errors=Exception):
    for attempt in range(tries):
        try:
            return call()
        except errors:
            # Reraise error on last attempt
            if attempt + 1 == tries:
                raise

Why use that?

There are obvious reasons: to reduce clutter and expose your intentions. However, there is usual cost. This as any abstraction brings an additional layer of complexity on top of python. And python could be seen as a layer on top of c, and that as one above asm, and that …

You can choose for yourself what level is too high. And if python is sometimes not enough high level language for you then you should definitely upgrade it.

I was going to continue this way and later introduce my library of a variety of functional tricks, but suddenly realized it’s not about FP, it’s about utility. And that’s what I will focus on below trying to show you real-life value of funcy.

Funcy started from a try to bunch up a couple of data manipulation utils. Therefore most of my examples will be about that. Some of them may seem trivial to you, but there are many gains these tiny tools can offer you both in terms of code brevity and expressiveness. Or they’ll just save your time, still good.

I’ll go through the typical tasks any python programmer face during her day.

Everyday data manipulation

1. Flatten list of lists. That’s how you usually do it:

1
2
3
4
5
6
7
8
9

from operator import concat
reduce(concat, list_of_lists)

# or that:
sum(list_of_lists, [])

# or that:
from itertools import chain
list(chain.from_iterable(list_of_lists))

They all work and all have their flaws: require imports, additional calls or restrict what you can pass into. But the main flaw is that they are all patterns not obvious calls. There should be a simple function to do such a simple and common thing and there is one in funcy:

1
2

from funcy import cat
cat(list_of_lists)

cat joins list of lists, tuples, iterators and generally any iterables into single list. And it comes with extra shortcut if you want to cat results of map call. For example, this

1
2

from funcy import mapcat
mapcat(str.splitlines, bunch_of_texts)

will result in flat list of all lines of all the texts. There are lazy versions of both functions: icat and imapcat.

2. Merge some dicts. There are several clumsy ways in python:

1
2
3
4
5

d1.update(d2)  # Changes d1
dict(d1, **d2) # Really awkward for more than 2 dicts

d = d1.copy()
d.update(d2)

I always wondered why one can’t just add them up, but that’s what we have. Anyway, this is also easy in funcy:

1
2
3
4

from funcy import merge, join
merge(d1, d2)
merge(d1, d2, d3)
join(sequence_of_dicts)

The best part here is these are omnivorous. They work with anything: sets, dicts, ordered dicts, lists, tuples, iterators, even strings, carefully preserving collection type.

3. Capturing something with regular expression. A usual way:

1
2
3
4

m = re.search(some_re, s)
if m:
    actual_match = m.group() # or m.group(i) or m.groups()
    ...

Much more straightforward with funcy:

1
2

from funcy import re_find
actual_match = re_find(some_re, s)

Still not impressed? Then look here:

1
2
3
4
5
6
7
8
9
10
11

from funcy import re_finder, re_all, partial, mapcat

# Get a number out of every word
map(re_finder(r'\d+'), words)

# Parse simple ini file into dict
# (re_finder returns tuples when there is more that one capture in regexp)
dict(imap(re_finder(r'(\w+)=(\w+)'), ini.splitlines()))

# Find all numbers in all the strings and return as flat list
mapcat(partial(re_all, r'\d+'), bunch_of_strings)

About imports and practicality

As you can see I import everything directly from funcy, not using any sub-packages. The reason it’s designed this way is practicality. That would be too annoying to remember where each one tiny thing comes from. There are enough libraries to clutter your file heads anyway.

This also enables you to write:

1

from funcy import *

And start enjoying all functional python niceness right away. Ok, now when you know where all the stuff is kept I won’t repeat imports in every new example.

A bit more functional things

We’ve seen a pair of examples of higher order functions earlier, particularly re_finder and partial. One thing to note is that re_finder itself is a partial application of re_find meant to be used with map and friends. Naturally, there is a similar utility to be used with filter:

1
2
3

# Choose all private attributes of an object
is_private = re_tester('^_')
filter(is_private, dir(some_obj))

We can create a bunch of predicates and filter with them:

1
2
3

is_special = re_tester('^__.+__$')
is_const = re_tester('^[A-Z_]+$')
filter(...)

But what if we want to apply several of them at once or use some predicate logic. That’s easy:

1
2
3

is_public = complement(is_private)
is_private_const = all_fn(is_private, is_const)
either_const_or_public = any_fn(is_const, is_public)

Or you can use convenience function complementary to filter:

1

remove(is_private, ...) # same as filter(is_public)

I hope everyone have their functional appetite satisfied so we can switch to something less abstract.

Collections

Aside from sequence utilities, funcy provides lots of ones to work with collections. And the two fundamental are walk and select, a versions of map and filter preserving collection type:

1
2
3
4
5
6
7
8
9
10
11
12
13

walk(inc, {1, 2, 3}) # -> {2, 3, 4}
walk(inc, (1, 2, 3)) # -> (2, 3, 4)

# Mapping function receives pairs when walking dicts
swap = lambda (k, v): (v, k)
walk(swap, {1: 10, 2: 20})
# -> {10: 1, 20: 2}

select(even, {1, 2, 3, 10, 20})
# -> {2, 10, 20}

select(lambda (k, v): k == v, {1: 1, 2: 3})
# -> {1: 1}

This pair of functions is backed up with a set of ones to work with dicts: walk_keys, walk_values, select_keys, select_values:

1
2
3
4
5

# Get a dict of public attributes of an instance
select_keys(is_public, instance.__dict__)

# Clean dict of falsy values
select_values(bool, some_dict)

The last example in this section will include several new functions at once: silent – catches all exceptions in passed function, returning None; compact – removes falsy values from collection; walk_values – maps dict values with given function. Anyhow, this line constructs a dict of integer params from typical stringy request dict you get:

1

compact(walk_values(silent(int), request_dict))

Back to data manipulation

Finally, the interesting part. I included some examples here just because they seem cool. Although, I did this earlier to be honest. Anyway, let’s split and group:

1
2
3
4
5

# split absolute and relative urls
absolute, relative = split(re_tester(r'^http://'), urls)

# group posts by category
group_by(lambda post: post.category, posts)