-
-
Notifications
You must be signed in to change notification settings - Fork 11
Hissp in Jupyter notebooks
Hissp's incremental, REPL-driven approach makes it a good fit for the notebook format.
The Hebigo prototype's REPL is implemented as a Jupyter kernel. This could work in a notebook as well.
Hissp compiles to Python, so there are a number of ways to run Hissp in the IPython kernel, which can be mixed with normal Python code.
If you're using Jupyterlite
and a Pyodide kernel,
you can install the stable version of Hissp from PyPI via micropip:
import micropip
await micropip.install('hissp')Hissp must (of course) be installed for the following to work.
Paste the following into an IPython cell (notebook or console):
from functools import reduce;import hissp;from IPython.core import magic
@magic.register_cell_magic # https://github.com/gilch/hissp/wiki/Hissp-in-Jupyter-notebooks
def lissp(_,c,r=hissp.reader.Lissp(env=vars())):return reduce(lambda _,x:hissp.evaluate(x),r.reads(c),None)Put a %%lissp line at the top of any cell with Lissp code in the same session.
If you don't have a better idea, start with the prelude:
%%lissp
hissp..prelude#:Click to try the %%lissp magic now.
Paste the following into an IPython cell:
get_ipython().input_transformers_cleanup.append(lambda L,r=__import__
('hissp') # https://github.com/gilch/hissp/wiki/Hissp-in-Jupyter-notebooks
.reader.Lissp(env=vars()):r.compile(''.join(L)).splitlines(keepends=1))Run it to make the Notebook language Lissp. Remember, you can still inject Python.
Beware that the compilation unit is the whole cell, not the individual top-level forms as in a .lissp file.
Click to try Lissp on Pyodide now.
IPython automatically strips the >>> and ... prompts produced by the interactive Python interpreter, meaning you can paste input lines from a session into notebook cells and they will still work.
The Lissp REPL also uses this format, and the Lissp #> and #.. prompts start with a #, which is the character for a Python line comment, so those lines will be ignored. That means you can paste your lines of Lissp input plus their following Python compilation sent to the Python interpreter from a LisspREPL session (including all the prompts) into a notebook cell running the IPython kernel and it should just work. Jupyter can also export such a notebook as a Python module, and that will also work.
You can run a Lissp cell via the lissp command-line tool as a subprocess by using the %%script lissp cell magic.
Globals won't be shared this way,
but you can see the output.
The typical use case is for scripts that only need to interact with the notebook via the filesystem,
for example, by recompiling a separate .lissp file to .py for import.
You can save the output or run the subprocess in the background and get it through a pipe.
See the %%script documentation for how.
Hissp need not even be installed in the notebook's environment for this to work. The magic could be an absolute path to a venv where the hissp package is installed. If you use this to invoke the transpiler and haven't broken Hissp's standalone property in your code, you should be able to import the resulting .py files even without a local hissp package install.
A form like %%script python -m hissp can use Python's other command-line options with Hissp, for example, using -O to disable assertions (and assure, __debug__, etc.)
Hissp code is Python data. When written mostly using Python's literal notation, this is called readerless mode.
You can use hissp.execute() to run Hissp forms in sequence.
This also returns their Python compilation as a string.
You can suppress a return value in IPython by appending a semicolon,
or by using some other final statement (like None) in the same cell.
While it's easy to explicitly print() any results you want to see,
Jupyter notebook cells also have the concept of an output value,
and may have rendering capabilities more advanced than plain text for certain object types (e.g., Pandas frames).
These are called rich outputs.
Call IPython.display.display() instead of print() to display a rich output in the Hissp code,
even if it's not the result of the cell. (This is normally added to the builtins for you by IPython.)
Once defined, code wrap magic can automatically add to a cell's code before running it.
You can use this to evaluate a readerless-mode form set to a magic global name
(FORM, in this case)
after executing the code in the cell.
For example:
%%code_wrap hissp
__code__
if 'FORM' in globals():
display(hissp.evaluate(FORM))
del FORM
__ret__This version doesn't do anything if FORM isn't set,
and unsets it after use.
Unlike hissp.execute(),
hissp.evaluate() can only evaluate a single form,
but its return value will be the result of evaluating the form,
rather than the compilation of the forms,
so we can display() it.
Example usage:
FORM = ('operator..add',1,2,)3
Multiple forms can be wrapped in a progn
(or a (('lambda',(),...,),), its expansion)
for use as a single top-level form.
The disadvantage of this approach is that top-level forms are compiled in one shot,
so any macro defined here won't take effect until the next top-level form,
since the entire form has already been compiled by the time the macro is defined.
But one can use multiple cells if that matters.
Simple cell magic is fairly easy to set up in IPython. The minimal implementation of a Lissp cell would be something like the following:
import hissp
from IPython.core import magic
@magic.register_cell_magic
def lissp(_line, cell, _reader=hissp.reader.Lissp(env=globals(), evaluate=True)):
_reader.compile(cell)Once evaluated in a cell, you can begin any cell with a %%lissp line and it will run the rest like a Lissp module.
The use of the globals() namespace means that global definitions are shared between the Lissp and Python cells.
The _reader is a mutable default argument on purpose. A Lissp object is supposed to count how many templates it's seen so it can give each gensym a unique hash, even if the code happens to be identical. It makes sense to reset this when compiling whole files for reproducible builds (each invocation of hissp.transpile() will create a new reader), but in a REPL or notebook environment, we should keep using the same one for the whole session. A global would also work here, but it pollutes the namespace and the magic would break if it's ever accidentally deleted. The magic function is the only thing using it, so it makes the most sense for it to live there.
IPython cell magic definitions can have a return value,
which becomes the result of the cell.
You're free to compute this in any way from the globals or the line or cell text.
If you prefer to have the result of the last top-level form become the cell output, which is similar to how the Python cells do it, the implementation is not much more difficult:
import hissp
from IPython.core import magic
@magic.register_cell_magic
def lissp(_line, cell, _reader=hissp.reader.Lissp(env=globals())):
last = None
for form in _reader.reads(cell):
last = hissp.evaluate(form)
return lastHere, we're reading one top-level form at a time into Hissp,
which we evaluate individually.
If you want to suppress a verbose output, add a None as the last form of the cell.
Note the use of hissp.evaluate() rather than hissp.execute().
This means you can't compile anything to a statement and expect it to work,
but Hissp normally compiles to expressions anyway.
If you need a statement for some reason,
you can't simply inject one at the top level as usual.
Instead, use a Python cell or call exec() yourself in the %%lissp cell.
Also note that the reader needs the env argument.
hissp.evaluate() has it as well, but it defaults to globals() if not specified via argument.
If you need them to use some other namespace, don't omit it.
See Defining custom magics in the IPython docs.
Note the recommendation about using %load_ext rather than registering magics on import directly with the decorator.
A magic defined outside the notebook can't simply use globals(),
as that refers to the defining module's, not the calling module's. Do this instead:
import hissp
from IPython.core import magic
_reader=hissp.reader.Lissp()
@magic.needs_local_scope
def lissp(_line, cell, local_env):
_reader.env = local_env
last = None
for form in _reader.reads(cell):
last = hissp.evaluate(form, local_env)
return lastWe don't have the env at definition time to put in the _reader, but the Lissp class has an env property you can set later. _reader could have been a default argument as before, but because this is in a separate module, there's no need.
Two main options:
%%lissp
hissp..prelude#:or
%%lissp
hissp..alias##H hissp.The prelude will dump a lot of things into the global environment. This is not usually a concern in a notebook environment, which is typically not meant to be a module imported elsewhere. IPython itself adds various things.
The alias approach is a bit cleaner but lacks the en- helpers and requires use of the H# tag.
You can still dump the prelude into some other namespace after aliasing:
%%lissp
(H#:define prelude H##:my my)
H##prelude (vars prelude)Pick a shorter name if you prefer.
An intermediate option (the REPL's default for __main__)
is to only copy the _macro_ namespace:
import copy
_macro_ = copy.copy(hissp._macro_)This won't add anything to the global namespace besides _macro_,
but still allows the use of the bundled macros unqualified.
Unlike the REPL, this cell magic doesn't show the Python compilation, although it may show up in tracebacks when an exception is raised.
You can display the Hissp form resulting from a Lissp expression by quoting it.
Applying pprint..pp to a complex form can make it easier to read.
Applying hissp..readerless to the form (the quoted Lissp expression)
shows its compilation without evaluating it.
If you find yourself doing this often enough, it might be worth modifying the cell magic:
import hissp
from IPython.core import magic
@magic.register_cell_magic
def lissp(line, cell, _reader=hissp.reader.Lissp(env=globals())):
import sys
last = None
for form in _reader.reads(cell):
python = hissp.readerless(form)
if line:
print(python, file=sys.stderr)
last = eval(python)
return lastNow you can add any line argument, like %%lissp ? and re-evaluate the cell to see its Python compilation as well.
Lissp compiles to Python, so an IPython notebook's default code cell language can be switched to Lissp by adding a transformation function to run the compiler to IPython's preprocessing hooks:
import hissp
def transform_lissp_to_python(lines, _reader=hissp.reader.Lissp(env=globals())):
return _reader.compile(''.join(lines)).splitlines(keepends=True)
get_ipython().input_transformers_cleanup.append(transform_lissp_to_python)Any cells run after this executes must be written in Lissp.
You can undo this and switch back to Python using
|get_ipython().input_transformers_cleanup.pop()|to remove the transform_lissp_to_python() function from the list of input transformers.
Remember that you can inject Python in Lissp.
IPython magics are a shorthand for get_ipython().run_cell_magic() or get_ipython().run_line_magic(). You can call these directly in Lissp, or write your own shorthand metaprograms using Lissp tags or Hissp macros, and they can expand to one of those form to use existing IPython magics.
Lissp can pass any lines through unmodified by wrapping them in |-| and it doesn't necessarily have to be valid Python. (Internal | should be escaped by doubling them.) A .lissp file can use this trick to compile to a .py file with a shebang line, for example. This can also be used to pass magics through to the cell text passed on to the IPython kernel, but they don't always work.
A slightly modified transformer could avoid preprocessing when there's a cell magic:
import hissp
def transform_lissp_to_python(lines, _reader=hissp.reader.Lissp(env=globals())):
if lines[0] == '%%IPython\n':
return lines[1:]
if lines[0].startswith('%%'):
return lines
return _reader.compile(''.join(lines)).splitlines(keepends=True)
get_ipython().input_transformers_cleanup.append(transform_lissp_to_python)The first if adds an %%IPython pseudo-cell magic to return the remainder of the cell's lines, which are processed as a normal IPython cell. It could also be followed by a normal cell magic on the next line, but the second if skips processing if the first line starts with %%, so this isn't required. You could write a version using either. If you only use the second (for cell magics), you could instead define a cell magic as normal to no-op for a normal IPython cell, or just use injections from Lissp when you need some Python.
Injected line magics should normally be written entirely in |-|. If you need to compile Lissp as an input for some reason, you could try using the mix macro.
A particularly useful magic is
%history -t
The -t is short for "transformed".
This shows a history of the fully expanded magics as well as the Python compilation when using transform_lissp_to_python().
IPython may execute the input transformers multiple times. Exactly how many times appears to be an implementation detail. This can result in compilation happening more than once. This is usually not a problem. Most Lissp tags are pure or at least idempotent, and this is recommended even without IPython. Hissp macros typically are as well, but there are occasionally legitimate uses for side effects in macros. If this is a problem in your notebook, you can suppress the side effects by using a cache:
import functools as ft, hissp
@lambda f: ft.update_wrapper(lambda lines: f(tuple(lines)), f)
@ft.lru_cache(maxsize=1)
def transform_lissp_to_python(lines, _reader=hissp.reader.Lissp(env=globals())):
return _reader.compile(''.join(lines)).splitlines(keepends=True)
get_ipython().input_transformers_cleanup.append(transform_lissp_to_python)The cache need only have size 1 to deal with the problem case of multiple compilations of the same input in a row. IPython passes in a list, which isn't hashable, so we need to transform it into a tuple before caching. That's what the extra decorator is for.
Beware that this will suppress the compile-time side effects even if you manually run the same cell twice in a row, which may be undesirable. But because the maximum cache size is 1, you can clear the cached code simply by compiling something else, either another cell or the same cell with any modification. You can fix this by using the length of the input history as an additional argument:
import functools as ft, hissp
@lambda f: ft.update_wrapper(lambda lines: f(tuple(lines), len(In)), f)
@ft.lru_cache(maxsize=1)
def transform_lissp_to_python(lines, _lenin, _reader=hissp.reader.Lissp(env=globals())):
return _reader.compile(''.join(lines)).splitlines(keepends=True)
get_ipython().input_transformers_cleanup.append(transform_lissp_to_python)The transform function doesn't use it; this just is so the cache will recognize it as a new input.
Caching like this opens another way of debugging compilation: by using a side effect.
import functools as ft, sys, hissp
@lambda f: ft.update_wrapper(lambda lines: f(tuple(lines)), f)
@ft.lru_cache(maxsize=1)
def transform_lissp_to_python(lines, _reader=hissp.reader.Lissp(env=globals())):
print(code:=_reader.compile(''.join(lines)), file=sys.stderr)
return code.splitlines(keepends=True)
get_ipython().input_transformers_cleanup.append(transform_lissp_to_python)This version will print the Python compilation to stderr for every cell by default, like the LisspREPL. Running the cell again will hide it (because it will use the cached value instead of running the function again).
While this all works in a notebook, using this approach in the IPython command-line REPL is not recommended. (Use the LisspREPL instead and edit code in your editor, or use one of the other solutions such as the %%lissp magic.) IPython will run the input transformers again for each line, which can result in a SoftSyntaxError from the Lissp reader. You could catch this in the transformation function. IPython will also avoid calling it for partial cells if you set the transformer's has_side_effects attribute to True, but IPython may not be able to tell when a Lissp cell is complete. The has_side_effects attribute seems to have no effect in notebooks.