Skip to content

Components

Lets consider a simple example component. All components inherit from Component:

# daiquiri/core/components/examplecomponent.py
# config in resources/config/example.yml
from marshmallow import Schema, fields

from daiquiri.core.components import Component, ComponentResource
from daiquiri.core.schema.component import ComponentSchema
from daiquiri.core.schema import ErrorSchema
from daiquiri.core import marshal


class ExampleConfigSchema(ComponentSchema):
    option = fields.Str(required=True)


class ExampleSchema(Schema):
    exampleid = fields.Int(required=True, metadata={"description": "The example id"})


class ExampleResource(ComponentResource):
    @marshal(
        out=[
            [200, ExampleSchema(), "An example output"]
            [400, ErrorSchema(), "Could not get example"]
        ],
    )

    def get(self, **kwargs):
        example = self._parent.a_method(kwargs["exampleid"])
        if example:
            return example
        else:
            return {"error": "Could not find example"}, 400


class Example(Component):
    _config_schema = ExampleConfigSchema()
    _config_export = []
    _actors = []

    _base_url = 'myexample'

    def setup(self, *args, **kwargs):
        option = self._config.get("option")
        self.register_route(ExampleResource, "/<int:exampleid>")

    def a_method(self, **kwargs):
        pass

Components get their configuation from the config directory in the server resources. The name should be lowercase class name. So for Example this should be example.yml. The resulting config is passed into the component as self._config as a dict. The mapping between component and config is defined in components.yml.

Optionally the config file can be marshalled by passing _config_schema. Config files that do not match the schema will halt the running application and dump an error message about the invalid fields.

self._config_export can be used to automatically pass keys from self._config to the UI via the /api/config resource. Sometimes it can be useful to access properties from the config in the UI.

Injected into components are the following core handlers as properties, i.e self._metadata:

Property Description
_hardware The hardware component, used to access hardware objects
_metadata The metadata component, used to interact with the database
_saving The saving component, used to create directories, setup data policy

Registering Routes

register_route tells the component that it wants to respond to a http requests and connects the Component to ComponentResources.

By default a components _base_url will be determined from the name of the component lower cased. In this example, by default it would be:

/api/example

By overriding _base_url a different root can be provided by the component. In the above example the component will now respond on

/api/myexample

Some examples:

self.register_route(ExampleResource, "") => /api/myexample
self.register_route(ExampleResource, "/resource") => /api/myexample/resource
self.register_route(ExampleResource, "/<int:exampleid>") => /api/myexample/5

Basic validators can also be added, in the last case the route will not respond unless the last parameter is an integer. See the flask documentation for available validators. Parmeters should generally be parsed by the marshaller, avoid using many url parameters to keep the api readable

Component Resources

ComponentResources allow the component to respond to http requests. Each component resource handles a series of http verbs. Input and output should always be marshalled as this cleans and validates user input, and makes sure output is consistent. Furthermore by using the marshal decorator API documentation is automatically generated for each resource.

Each method in a ComponentResource recieves all of the http arguments (post, query, body) into **kwargs and the parent component into self._parent.

The default response code is 200, additional response code can be returned as shown below, for example a 400 if the example cannot be found

from daiquiri.core.components import Component, ComponentResource
from daiquiri.core.schema import ErrorSchema
from daiquiri.core import marshal


class ExamplesResource(ComponentResource):
    @marshal(
        inp=ExampleSchema,
        out=[
            [200, ExampleSchema(), "An example output"]
            [400, ErrorSchema(), "Could not get example"]
        ],
    )

    def post(self, **kwargs):
        example = self._parent.a_method(kwargs.get("limit"))

        if example:
            return example
        else:
            return {"error": "Could not find example"}, 400

    @marshal(
        ...
    )
    def get(self, **kwargs):
        pass


class Example(Component):
    def setup(self, *args, **kwargs):
        self.register_route(ExamplesResource, "")

Marshalling and Schemas

All input and output is marshalled using the @marshal decorator:

class ExamplesResource(ComponentResource):
    @marshal(
        inp=ExampleSchema,
        out=[
            [200, ExampleSchema(), "An example output"]
            [400, ErrorSchema(), "Could not get example"]
        ],
    )
    def post(self, **kwargs):
        pass

The inp keyword defines how the input should be marshalled, this can take either a schema, or a dict of marshmallow fields.

Using a schema:

@marshal(
    inp=ExampleSchema,
)
...

As a dict:

@marshal(
    inp={
        "limit": fields.Int(), 
        "offset": fields.Int(metadata={"description": "Offset to start records at"})
    },
)
...
A dict of fields should be used sparingly, and is generally used to accept additional query parameters, for example, for paging a list of data collections. A description keyword can be passed to document the field, this will end up in the documentation.

The out keyword tells the marshaller how to format the output, this takes a list of lists, each sublist contains a http status_code, a Schema, and a short documentation string of what the response is. This will be used in the auto generated API documentation

@marshal(
    out=[
        [200, ExampleSchema(), "An example output"]
        [400, ErrorSchema(), "Could not get example"]
    ],
)
...

Marshalling makes use of the python module marshmallow. See the marshmallow documentation for a full list of fields that are avaialble. A host of custom fields are provided in daiquiri.core.schema.validators, the ValidatedRegexp validator should be used to enforce strong control of accepted string types. fields.Str() is generally not sufficient for string validation, be more explicit.

Schemas should be stored in their own file, i.e. daiquiri.core.schema.example, and the description keyword should be used to document what the field is. See the marshmallow documentation for more details on writing Schemas.

from marshmallow import Schema, fields


class ExampleSchema(Schema):
    exampleid = fields.Int(required=True, metadata={"description": "The example id"})

Some common Schemas are provided to ensure consistent errors and general messages:

MessageSchema:

from daiquiri.core.schema import MessageSchema

...
@marshal(
    out=[[200, MessageSchema(), "Could not get example"]],
)
def post(self, **kwargs):
    return {"message": "im a message"}

ErrorSchema:

from daiquiri.core.schema import ErrorSchema

...
@marshal(
    out=[[400, ErrorSchema(), "Could not get example"]],
)
def post(self, **kwargs):
    return {"error": "im an error message, with http status code"}, 400

Common input parameters

The @marshal decorator can auto configure the ComponentResource to parse input parameters for a number of standard cases. Making use of these keeps input parameters consistent across resources. Pass the following keywords with bool True to the @marshal decorator

Keyword Fields Description
paged page (int), per_page (int) To parse the current page and number of results per page
ordered order (str), order_by (str) To parse ordering parameters, order column to order by order_by the sort order: asc or desc
filtered s (str) Enable the s parameter for searching
@marshal(
    inp=ExampleSchema,
    paged=True,
    ordered=True,
    filtered=True
)

Pagination

Resources that provide many rows should be paginated in order to keep http response size reasonable. Wrapper the relevant schema with paginated:

from daiquiri.core.schema.metadata import paginated


...
@marshal(
    out=[[200, paginated(DataCollectionSchema), "List of datacollections"]],
)

For the above example, this will now enforce a response of the format:

{
    "total": 25,
    "rows": [DataCollectionSchema, DataCollectionSchema, ...]
}

total should contain the total number of rows available, rows should contain the current slice.

Decorators

Daiquiri provides a few useful decorators for ComponentResources:

Ensure the user has control before running this resource:

from daiquiri.core import require_control


class ExamplesResource(ComponentResource):
    @require_control
    def post(self, **kwargs):
        pass

Ensure the user is a staff member before running this resource:

from daiquiri.core import require_staff


class ExamplesResource(ComponentResource):
    @require_staff
    def post(self, **kwargs):
        pass

SocketIO

As well as creating REST resources a component can also send and recieve websocket requests via SocketIO, two convience methods are provided to components.

A namespace is created automatically for each component with the same prefix as the _base_url.

emit allows components to emit a SocketIO message:

self.emit(
    "message",
    payload,
)

on allows a component to respond to a SocketIO message (much less frequent use case)

def setup(self):
    self.on("input")(self.input)

...

def input(self, payload):
    pass

The namespace can be overriden via:

class Example(Component):
    _namespace = "mynamespace"

Actors

Components can create and launch Actors which are defined in the implementors python module specified in the component's config file. The actors of a component are in a subdirectory whose name is the lowercase class name. So for Example this would be implementors/example.

Required actors for a component should be defined in self._actors:

_actors = ['actor1', 'actor2']

This tells the component that two actors are available to it, they must then be mapped to the relevant implementor files and classes via the component config file actors key:

actors:
  actor1: actor1
  actor2: actor2

This allows flexibility in mapping what the component requires and where the files live, generally the key and value will be the same. Daiquiri will resolve the value to the example subdirectory in the implementors module i.e. implementors.example.<value>. Filenames should be lowercase and classes should start with an uppercase letter.

Module Resolution

The loader expects the daiquiri classname in lower case. It will resolve to, for example:

  • actor1 -> daiquiri.implementors.example.actor1.Actor1
  • actor2 -> daiquiri.implementors.example.actor2.Actor2
  • actor2 -> daiquiri_id00.implementors.example.actor2.Actor2

Once registered actors can be launched as follows:

actid = self.actor(
    "actor1",
    spawn=True,
    success=self._actor_finished,
    error=self._actor_failed,
    actargs={
        ...
    },
)

The return value is a unique hash for this actor.

The following keywords are accepted:

Key Type Default Description
spawn bool false Whether to launch this actor in its own greenlet, otherwise it will be placed into the queue
enqueue bool false Whether to put the actor in the queue, if spawn and enqueue are false the actor is placed at the top of the queue
start func None A callback when the actor starts
success func None A callback if the actor succeeds
error func None A callback if the actor fails
remove func None A callback if the actor is removed from the queue
actargs dict {} A dict of parameters passed to the actor

The callbacks have the following signatures:

def start(self, actid, actor):
    ...

def success(self, actid, return_value, actor):
    ...

def error(self, actid, exception, actor):
    ...

def remove(self, actid, actor):
    ...

Actor Resources

An @actor decorator is provided that can automap a ComponentResource to launch an actor.

The simplest case will just launch an actor with the keywords from the request. User input will be validated according the the Actor Schema:

class ActorResrouce(ComponentResource):
    @actor("actor1", enqueue=True, [...**actargs])
    def actor(self, **kwargs):
        pass

The second case with the preprocess keyword allows for preprocessing of the user input by a secondary preprocess function. This can add extra parameters and pass these into the actor. The function should return the updated **kwargs.

A more complex example:

class MosaicResource(ComponentResource):
    @require_control
    @actor("mosaic", enqueue=False, preprocess=True)
    def post(self, **kwargs):
        """Create a tiled mosaic actor"""
        pass

    def preprocess(self, **kwargs):
        kwargs["absol"] = self._parent.get_absolute_fp(
            {"x": kwargs["x1"], "y": kwargs["y1"]},
            {"x": kwargs["x2"], "y": kwargs["y2"]},
        )

        sample = self._metadata.get_samples(sampleid=kwargs["sampleid"])
        if not sample:
            raise AttributeError(f"No such sample {kwargs['sampleid']}")

        sessionid = g.blsession.get("sessionid")

        def save_image(x, y):
            return self._parent.save_image(
                sessionid=sessionid,
                sampleid=kwargs["sampleid"],
                file_prefix=f"mosaic_{x}_{y}_",
            )

        kwargs["sessionid"] = sessionid
        kwargs["save"] = save_image

        return kwargs

Actor decorator args:

Keyword Description
enqueue Place the actor in the queue
spawn Spawn the actor immediately in a new greenlet
synchronous Spawn the actor immediately in a new greenlet, and await its termination, returning its value in result

Accessing Components

Components can request other components via:

self.get_component("component")

i.e. a component could get the scans component to retrive scan data:

scans = self.get_component("scans")
scalars = scans.get_scan_data(scanid=dc["datacollectionnumber"], per_page=1e10)

This should be used sparingly as it couples components together.

Flask request context

The flask request context g is populated with some useful things during a request (not valid outside of a http request!):

from flask import g

g.user

Contains an instance of [daiquiri.core.metadata.user][] for the current user. This allows to check for example if the user is a staff member:

g.user.staff() => bool 

Or ask which permissions the user has:

g.user.permission("super_admin") => bool

g.blsession

Contains a dict of the currently selected session from the metadata handler:

g.blsession.get("sessionid") => int

g.blsession.get("proposal") => str