I want to record each change to the program’s state, so I take the original
source code and add logging calls around each assignment, loop iteration,
and other interesting features. Each change I make to the code needs to leave
its behaviour the same while recording enough information to build a display.
The recording is done by making calls to a new global variable:
As an example, consider the following code with its display:
a = 'Hello' | a = 'Hello' b = a + ', World!' | b = 'Hello, World!'
That code could be instrumented like this:
a = __live_coding_context__.assign('a', 'Hello', 1) b = __live_coding_context__.assign('b', a + ', World!', 2)
assign() method takes in the calculated value, records the assignment
for its report, and then returns the value for the actual assignment.
The numbers are the line numbers from the original code, and they tell the
context where to report the assignments so they show up on the right lines.
To display loop execution, I add the pipe symbol at the end of every line in the
loop, each time the loop starts. The context has a
start_block() method to
do this. Here’s an example with its display:
x = 0 | x = 0 for n in [1, 3, 5]: | n = 1 | n = 3 | n = 5 x = x + n | x = 1 | x = 4 | x = 9
That could be instrumented like this, including
instrumentation of the loop variable:
x = __live_coding_context__.assign('x', 0, 1) for n in [1, 3, 5]: __live_coding_context__.start_block(2, 3) __live_coding_context__.assign('n', n, 2) x = __live_coding_context__.assign('x', x + n, 3)
Once I know what changes I want to make to the code, how do I add things to
the user’s source code? Python has an amazing module called
ast for handling
abstract syntax trees. You can parse Python source code from a string into a
syntax tree, and then make changes to it. Finally, you can compile and run it.
Here’s some example source code:
s = 'Hello' s += ', World!'
Here are the
ast objects that it gets parsed into:
Module(body=[Assign(targets=[Name(id='s', ctx=Store())], value=Str(s='Hello')), AugAssign(target=Name(id='s', ctx=Store()), op=Add(), value=Str(s=', World!'))])
Those objects also have attributes holding the line numbers they came from, so I can display the assignments next to the matching lines of source code. To make changes to an abstract syntax tree, you define a class that visits each node of the tree, and returns the modified node. Here’s a simple example, the method that modifies a for loop:
def visit_For(self, node): new_node = self.generic_visit(node) line_numbers = set() self._find_line_numbers(new_node, line_numbers) args = [Num(n=min(line_numbers)), Num(n=max(line_numbers))] new_body = [self._create_context_call('start_block', args)] new_body.extend(self._trace_assignment_list(new_node.target)) new_body.extend(new_node.body) new_node.body = new_body return new_node
generic_visit() visits all the child nodes before modifying the for
loop itself. Next, I walk through the child nodes looking for all the line
numbers that are included in the loop, so I can draw the pipe symbols between
each iteration of the loop. Then I insert calls to
assign() before the original statements in the for loop. Finally, I return
the modified for loop.
One of the most challenging parts was to register the instrumented code as a module so a developer can register callbacks to the instrumented code. This is useful for running unit tests, for example. David Beazley wrote an exhaustive review of Python’s module system, and that showed how to manually override the import process.
Here’s a minimal example of how to replace the standard import:
from ast import parse import types import sys source = """\ def foo(): print('Hello, World!') """ module_name = 'live_lib' tree = parse(source) # TODO: Instrument tree. code = compile(tree, '<dummy file>', 'exec') mod = types.ModuleType(module_name) sys.modules[module_name] = mod exec(code, mod.__dict__) import live_lib live_lib.foo()
The source string holds the original source code, and I parse that into a
syntax tree. After instrumenting the tree (not shown), I compile it and execute
it using the module’s dictionary for its global variables. After that, importing
live_lib module will import the instrumented code, even if there is no
live_lib.py file. Any other modules that import
live_lib will get the
I also learned enough about Python’s module system to replace the regular
turtle with my mock turtle. Now I don’t need any special variables or method
calls to visualize turtle graphics. The exact same code can run in live coding
mode or in regular mode. The technique of replacing some framework code with
a new version is called monkey patching; here’s the
@classmethod def monkey_patch(cls, canvas=None): turtle_module = sys.modules['turtle'] turtle_module.Turtle = MockTurtle turtle_module.mainloop = lambda: None MockTurtle._screen = MockTurtle._Screen(canvas) MockTurtle._pen = MockTurtle()
This replaces the regular
turtle.Turtle class with
MockTurtle, as well
as registering an instance of MockTurtle that will record all its commands.
mainloop() method usually keeps the turtle window open, but the
replacement does nothing, because there is no window to keep open.