Embedding Python and importing from memory

Time to ask the web again:

In Glom, I use PyRun_String() to execute python scripts from memory (the scripts are never on disk), and get the result. But I’d like those scripts to be able to import Python modules that are also in memory (in a virtual library of reusable code). Python’s import statement usually just looks for files in the Python import path. I’ve looked through the Python/C API reference, but I can’t see anything suitable. Unlike C, you can’t just paste the code into the start of the script, because being in a module affects the syntax.

I have a vague idea that PyImport_ExecCodeModule might make the Python code importable under the provided name, but it seems to require a compiled object, for which I need to supply a filename to PyNode_Compile().

I’d rather not write all the files to a temporary directory just so they can be imported. That seems like a fragile hack.

Update:

object = Py_CompileString(script_text, module_name, Py_File_Input) followed by PyImport_ExecCodeModule(module_name, object) seems to work. Thanks commenters. But now I wonder how to remove modules when they have been removed from the virtual library. Maybe I could do PyImport_Cleanup() each time, but that is documented as for internal use only.

10 thoughts on “Embedding Python and importing from memory

  1. Since python doesn’t re-import modules if they have already been imported, it seems like you should be able to inject them into the global module dictionary from your C code.

    Hence, I think you could use Py_CompileString to compile the module and then PyImport_ExecCodeModule, no?

  2. Ian, I think that PEP led to something actually going into Python, but I’m not sure what. I should investigate this imputil thing that seems to be in Python now.

    Johannes, I thought of that already, but Py_CompileString() takes a filename, not the file contents.

  3. Python Advisor,

    that’s an interesting approach. Thanks. With what should I “fill” the empty object that is returned? Just a python string?

  4. My reading of the Python C API documentation indicates that there are two ways to achieve what you want:

    The first way is to use Py_CompileString(“your code here”, “mymodule1”, Py_file_input) and pass the result to
    PyImport_ExecCodeModule(“mymodule1”).

    The second way is to use Py_AddModule(“mymodule1”), call PyModule_GetDict on the result, and fill in that dict with your global functions, classes and variables. This way seems to be rather inconvenient if you do not already have the compiled function objects.

  5. I recently came across your blog and have been reading along. I thought I would leave my first comment. I don’t know what to say except that I have enjoyed reading. Nice blog. I will keep visiting this blog very often.

    Maria

    http://memory1gb.com

Comments are closed.