I want to record each change to the program’s state, so I take the original
source code and add logging calls around each assignment, loop iteration,
and every other interesting feature. Each change I make to the code needs to leave
its behaviour the same while recording enough information to build a display.
The recording is done by making calls to a new global variable:
As an example, consider the following code with its display:
a = 'Hello' | a = 'Hello' b = a + ', World!' | b = 'Hello, World!'
That code could be instrumented like this:
a = __live_coding_context__.assign('a', 'Hello', 1) b = __live_coding_context__.assign('b', a + ', World!', 2)
assign() method takes in the calculated value, records the assignment
for its report, and then returns the value for the actual assignment.
The numbers are the line numbers from the original code, and they tell the
context where to report the assignments so they show up on the right lines.
To display loop execution, I add the pipe symbol at the end of every line in the
loop, each time the loop starts. The context has a
start_block() method to
do this. Here’s an example with its display:
x = 0 | x = 0 for n in [1, 3, 5]: | n = 1 | n = 3 | n = 5 x = x + n | x = 1 | x = 4 | x = 9
That could be instrumented like this, including
instrumentation of the loop variable:
x = __live_coding_context__.assign('x', 0, 1) for n in [1, 3, 5]: __live_coding_context__.start_block(2, 3) __live_coding_context__.assign('n', n, 2) x = __live_coding_context__.assign('x', x + n, 3)
Once I know what changes I want to make to the code, how do I add things to
the user’s source code? Python has an amazing module called
ast for handling
abstract syntax trees. You can parse Python source code from a string into a
syntax tree, and then make changes to it. Finally, you can compile and run it.
Here’s some example source code:
s = 'Hello' s += ', World!'
Here are the
ast objects that it gets parsed into:
Module(body=[Assign(targets=[Name(id='s', ctx=Store())], value=Str(s='Hello')), AugAssign(target=Name(id='s', ctx=Store()), op=Add(), value=Str(s=', World!'))])
Those objects also have attributes holding the line numbers they came from, so I can display the assignments next to the matching lines of source code. To make changes to an abstract syntax tree, you define a class that visits each node of the tree, and returns the modified node. Here’s a simple example, the method that modifies a for loop:
def visit_For(self, node): new_node = self.generic_visit(node) line_numbers = set() self._find_line_numbers(new_node, line_numbers) args = [Num(n=min(line_numbers)), Num(n=max(line_numbers))] new_body = [self._create_context_call('start_block', args)] new_body.extend(self._trace_assignment_list(new_node.target)) new_body.extend(new_node.body) new_node.body = new_body return new_node
generic_visit() visits all the child nodes before modifying the for
loop itself. Next, I walk through the child nodes looking for all the line
numbers that are included in the loop, so I can draw the pipe symbols between
each iteration of the loop. Then I insert calls to
assign() before the original statements in the for loop. Finally, I return
the modified for loop.
One of the most challenging parts was to register the instrumented code as a module so a developer can see calls into the instrumented code from another module. This is useful for running unit tests, for example. David Beazley wrote an exhaustive review of Python’s module system, and that showed how to manually override the import process.
Here’s a minimal example of how to replace the standard import:
from ast import parse import types import sys source = """\ def foo(): print('Hello, World!') """ module_name = 'live_lib' tree = parse(source) # TODO: Instrument tree. code = compile(tree, '<dummy file>', 'exec') mod = types.ModuleType(module_name) sys.modules[module_name] = mod exec(code, mod.__dict__) import live_lib live_lib.foo()
The source string holds the original source code, and I parse that into a
syntax tree. After instrumenting the tree (not shown), I compile it and execute
it using the module’s dictionary for its global variables. After that, importing
live_lib module will import the instrumented code, even if there is no
live_lib.py file. Any other modules that import
live_lib will get the
I also learned enough about Python’s module system to replace the regular
turtle with my mock turtle. Now I don’t need any special variables or method
calls to visualize turtle graphics. The exact same code can run in live coding
mode or in regular mode. The technique of replacing some framework code with
a new version is called monkey patching; here’s the
@classmethod def monkey_patch(cls, canvas=None): turtle_module = sys.modules['turtle'] turtle_module.Turtle = MockTurtle turtle_module.mainloop = lambda: None MockTurtle._screen = MockTurtle._Screen(canvas) MockTurtle._pen = MockTurtle()
This replaces the regular
turtle.Turtle class with
MockTurtle, as well
as registering an instance of MockTurtle that will record all its commands.
mainloop() method usually keeps the turtle window open, but the
replacement does nothing, because there is no window to keep open.
Both of those features work better if you can wait until the user’s code imports a module before replacing it with a traced or monkey patched version. For example, if I always imported matplotlib and monkey patched it, it would slow down scripts that don’t use matplotlib. To detect when a module is being imported, I added two new import hooks to the meta path.
The main classes are:
Tracervisits the abstract syntax tree for a module, and adds all the tracing calls.
TraceRunneris a main control class that coordinates
Tracerand the import hooks.
TracedModuleImporteris an import hook that decides which module should have the tracing calls added when it is imported. It’s an importer, which means that it is both a finder and a loader.
PatchedModuleFinderis an import hook that decides which modules should be monkey patched when they are imported. This delegates the basic loading to other entries in the meta path, and then patches the resulting module.
PatchedModuleLoaderis the loader for the
FileSwallowerreports calls to
stderr.write()and other output.
write()calls on local variables that hold
Canvasrecords calls to drawing methods, usually from the
MockTurtlereplaces the regular
Turtle, and records all its commands.
Windowclass, and displays its interface on the preview canvas.
ReportBuilderrecords all of the assignments and loop events, then displays them in a text report.
DeletionTargetwraps an object before it has an item or attribute deleted from it. If the object’s
repr()changes, then it records an assignment event.
TreeVisitorparse the source code for a new module during the import process, and check which methods or classes have been selected for tracing.