Fetcher Function

The fetcher function is used to fetch resources from a given file or URL.

Configuration

A new function of type fetcher named <id> can be created as follows:

mldb.put("/v1/functions/"+<id>, {
    "type": "fetcher",
    "params": {
        "maxConcurrentFetch": <int>
    }
})

with the following key-value definitions for params:

Field, Type, DefaultDescription

maxConcurrentFetch
int
-1

The maximum number of concurrent fetching operations to allow. -1 leaves the control to MLDB.

Input and Output Values

The function takes a single input named url that contains the URL of the resource to load.

It will return two output columns:

Example

The following Javascript creates and calls a function that will return the country code from an IP address from an external web service.

mldb.put("/v1/functions/fetch", { "type": "fetcher" })
SELECT CAST (fetch({url: 'http://www.google.com'})[content] AS STRING)

Limitations

Design notes

The fetcher function is a function, and not a builtin, to allow configuration of authentication, caching and other parameters in a future release to address the limitations above.