Python Plugins

This plugin allows for a plugin that is implemented in Python to be loaded into MLDB to extend its functionality.

Configuration

A new plugin of type python named <id> can be created as follows:

mldb.put("/v1/plugins/"+<id>, {
    "type": "python",
    "params": {
        "address": <string>,
        "source": <PackageElementSources>,
        "args": <Any>
    }
})

with the following key-value definitions for params:

Field, Type, DefaultDescription

address
string

URI or location of script to run (use this parameter OR 'source')

source
PackageElementSources

source of script to run (use this parameter OR 'address')

args
Any

Arguments to pass to script

With PackageElementSources defined as:

Field, Type, DefaultDescription

main
string

source for the main element of the plugin

routes
string

source for the routes element of the plugin

status
string

source for the status element of the plugin

If the address parameter is used, it may contain:

If the source parameter is used, main will be executed to initialize the plugin and routes will be executed to handle REST requests (see mldb.plugin.rest_params below)

If the args parameter is used, its contents are available to the main Python code via the mldb.plugins.args variable (see below).

Server-side Python API

Plugins and scripts running within the MLDB instance will have access to an mldb object which lets them manipulate the database more efficiently than via HTTP network round-trips.

mldb object (available to plugins and scripts)

Filesystem access

There are two functions that allow access to the virtual filesystem of MLDB:

dataset object (available to plugins and scripts)

mldb.script object (available to scripts)

mldb.plugin object (available to plugins)

Handling a custom route

Calling /v1/plugins/<id>/routes/<route> will trigger the execution of the code in routes.py. The plugin developer must handle the (verb, remaining) tuple, available in the mldb.plugin.rest_params object. If it represents a valid route, the set_return function must be called with a non-null body, which will be returned in the response. If the function is not called or called with a null body, the HTTP response code will be 404.

Plugin output

GET /v1/plugins/<id>/routes/lastoutput will return the output of the latest plugin code to have run: either the plugin loader or any subsequent calls to request handlers (see mldb.plugin.set_request_handler() above).

Script output will be returned as the HTTP response.

In either case, the output will look like this:

{
    "logs": [<logs>], 
    "result": <return_value>,
}

where <logs> come from calls to mldb.log() or using the python print statement to standard output or standard error, and <return_value> comes from a call to mldb.set_return() (see above).

If an exception was thrown, the output will look like

{
    "logs": [<logs>], 
    "exception": <exception details>,
}

The actual output object looks like this:

Field, Type, DefaultDescription

result
JSON

Result of running script

logs
ARRAY [ ScriptLogEntry ]

Log entries created by script

exception
LINK std::shared_ptr

Exception thrown by script

extra
Any

Extra information from language

with log entries looking like

Field, Type, DefaultDescription

ts
Date
"1970-01-01T00:00:00Z"

Timestamp at which message was received

s
string

Stream on which message was received

c
ScriptLogContent

Content of stream

closed
bool
false

Stream is closed

Exceptions are represented as

Field, Type, DefaultDescription

message
string

Exception description

where
string

Full description of where exception came from

scriptUri
string

URI of script that caused the exception

lineNumber
int
-1

Number of the line that caused the exception

columnStart
int
-1

Start column in the line

columnEnd
int
-1

End column in the line

lineContents
string

Contents of the line indicating where the exception was

context
ARRAY [ string ]

What we were doing when we threw the exception

stack
ARRAY [ ScriptStackFrame ]

Call stack for exception

extra
Any

Extra information from language about exception

with stack frames like

Field, Type, DefaultDescription

scriptUri
string

URI of script in this frame

functionName
string

Name of function in this frame

where
string

Where is the frame, in natural format for language

lineNumber
int
-1

Line number

columnStart
int
-1

Column number of error

columnEnd
int
-1

End column number of error

extra
Any

Extra stack from information from language

Note that the Python plugin only fills in the where field of stack frame entries.

Debugging

The following are useful for debugging MLDB, but should not be used in normal use of MLDB:

mldb_wrapper

A higher level object named mldb_wrapper can be used to wrap the mldb object. Key features are:

Example

Code ```python

We create the mldb_wrapper object

wrapper = mldb_wrapper.wrap(mldb)

The functions of the mldb object are still accessible. Here we use

create_dataset, record 2 rows and commit.

ds = wrapper.createdataset({'id' : 'ds', 'type' : 'sparse.mutable'}) ds.recordrow('row 1', [['ColA', 'A', 0]]) ds.record_row('row 2', [['ColB', 'B', 0]]) ds.commit()

We use the query function which returns a list of the rows.

res = wrapper.query("SELECT * FROM ds")

Output the result. Since it's a list the log function will format it.

wrapper.log(res)

The way to set the return is the same as before. Here we set "success".

mldb.script.set_return("success")


Output
```
2016-10-19 20:37:34.535 script runner plugin [
    [
        "_rowName",
        "ColB",
        "ColA"
    ],
    [
        "row 2",
        "B",
        null
    ],
    [
        "row 1",
        null,
        "A"
    ]
]
```

For more examples, refer to the python tests, which are mostly written with the `mldb_wrapper`.