Components
Lets consider a simple example component. All components inherit from Component:
# daiquiri/core/components/examplecomponent.py
# config in resources/config/example.yml
from marshmallow import Schema, fields
from daiquiri.core.components import Component, ComponentResource
from daiquiri.core.schema.component import ComponentSchema
from daiquiri.core.schema import ErrorSchema
from daiquiri.core import marshal
class ExampleConfigSchema(ComponentSchema):
option = fields.Str(required=True)
class ExampleSchema(Schema):
exampleid = fields.Int(required=True, metadata={"description": "The example id"})
class ExampleResource(ComponentResource):
@marshal(
out=[
[200, ExampleSchema(), "An example output"]
[400, ErrorSchema(), "Could not get example"]
],
)
def get(self, **kwargs):
example = self._parent.a_method(kwargs["exampleid"])
if example:
return example
else:
return {"error": "Could not find example"}, 400
class Example(Component):
_config_schema = ExampleConfigSchema()
_config_export = []
_actors = []
_base_url = 'myexample'
def setup(self, *args, **kwargs):
option = self._config.get("option")
self.register_route(ExampleResource, "/<int:exampleid>")
def a_method(self, **kwargs):
pass
Components get their configuation from the config directory in the server resources. The name should be lowercase class name. So for Example this should be example.yml. The resulting config is passed into the component as self._config as a dict. The mapping between component and config is defined in components.yml.
Optionally the config file can be marshalled by passing _config_schema. Config files that do not match the schema will halt the running application and dump an error message about the invalid fields.
self._config_export can be used to automatically pass keys from self._config to the UI via the /api/config resource. Sometimes it can be useful to access properties from the config in the UI.
Injected into components are the following core handlers as properties, i.e self._metadata:
| Property | Description |
|---|---|
| _hardware | The hardware component, used to access hardware objects |
| _metadata | The metadata component, used to interact with the database |
| _saving | The saving component, used to create directories, setup data policy |
Registering Routes
register_route tells the component that it wants to respond to a http requests and connects the Component to ComponentResources.
By default a components _base_url will be determined from the name of the component lower cased. In this example, by default it would be:
By overriding _base_url a different root can be provided by the component. In the above example the component will now respond on
Some examples:
self.register_route(ExampleResource, "") => /api/myexample
self.register_route(ExampleResource, "/resource") => /api/myexample/resource
self.register_route(ExampleResource, "/<int:exampleid>") => /api/myexample/5
Basic validators can also be added, in the last case the route will not respond unless the last parameter is an integer. See the flask documentation for available validators. Parmeters should generally be parsed by the marshaller, avoid using many url parameters to keep the api readable
Component Resources
ComponentResources allow the component to respond to http requests. Each component resource handles a series of http verbs. Input and output should always be marshalled as this cleans and validates user input, and makes sure output is consistent. Furthermore by using the marshal decorator API documentation is automatically generated for each resource.
Each method in a ComponentResource recieves all of the http arguments (post, query, body) into **kwargs and the parent component into self._parent.
The default response code is 200, additional response code can be returned as shown below, for example a 400 if the example cannot be found
from daiquiri.core.components import Component, ComponentResource
from daiquiri.core.schema import ErrorSchema
from daiquiri.core import marshal
class ExamplesResource(ComponentResource):
@marshal(
inp=ExampleSchema,
out=[
[200, ExampleSchema(), "An example output"]
[400, ErrorSchema(), "Could not get example"]
],
)
def post(self, **kwargs):
example = self._parent.a_method(kwargs.get("limit"))
if example:
return example
else:
return {"error": "Could not find example"}, 400
@marshal(
...
)
def get(self, **kwargs):
pass
class Example(Component):
def setup(self, *args, **kwargs):
self.register_route(ExamplesResource, "")
Marshalling and Schemas
All input and output is marshalled using the @marshal decorator:
class ExamplesResource(ComponentResource):
@marshal(
inp=ExampleSchema,
out=[
[200, ExampleSchema(), "An example output"]
[400, ErrorSchema(), "Could not get example"]
],
)
def post(self, **kwargs):
pass
The inp keyword defines how the input should be marshalled, this can take either a schema, or a dict of marshmallow fields.
Using a schema:
As a dict:
@marshal(
inp={
"limit": fields.Int(),
"offset": fields.Int(metadata={"description": "Offset to start records at"})
},
)
...
description keyword can be passed to document the field, this will end up in the documentation.
The out keyword tells the marshaller how to format the output, this takes a list of lists, each sublist contains a http status_code, a Schema, and a short documentation string of what the response is. This will be used in the auto generated API documentation
@marshal(
out=[
[200, ExampleSchema(), "An example output"]
[400, ErrorSchema(), "Could not get example"]
],
)
...
Marshalling makes use of the python module marshmallow. See the marshmallow documentation for a full list of fields that are avaialble. A host of custom fields are provided in daiquiri.core.schema.validators, the ValidatedRegexp validator should be used to enforce strong control of accepted string types. fields.Str() is generally not sufficient for string validation, be more explicit.
Schemas should be stored in their own file, i.e. daiquiri.core.schema.example, and the description keyword should be used to document what the field is. See the marshmallow documentation for more details on writing Schemas.
from marshmallow import Schema, fields
class ExampleSchema(Schema):
exampleid = fields.Int(required=True, metadata={"description": "The example id"})
Some common Schemas are provided to ensure consistent errors and general messages:
MessageSchema:
from daiquiri.core.schema import MessageSchema
...
@marshal(
out=[[200, MessageSchema(), "Could not get example"]],
)
def post(self, **kwargs):
return {"message": "im a message"}
ErrorSchema:
from daiquiri.core.schema import ErrorSchema
...
@marshal(
out=[[400, ErrorSchema(), "Could not get example"]],
)
def post(self, **kwargs):
return {"error": "im an error message, with http status code"}, 400
Common input parameters
The @marshal decorator can auto configure the ComponentResource to parse input parameters for a number of standard cases. Making use of these keeps input parameters consistent across resources. Pass the following keywords with bool True to the @marshal decorator
| Keyword | Fields | Description |
|---|---|---|
| paged | page (int), per_page (int) | To parse the current page and number of results per page |
| ordered | order (str), order_by (str) | To parse ordering parameters, order column to order by order_by the sort order: asc or desc |
| filtered | s (str) | Enable the s parameter for searching |
Pagination
Resources that provide many rows should be paginated in order to keep http response size reasonable. Wrapper the relevant schema with paginated:
from daiquiri.core.schema.metadata import paginated
...
@marshal(
out=[[200, paginated(DataCollectionSchema), "List of datacollections"]],
)
For the above example, this will now enforce a response of the format:
total should contain the total number of rows available, rows should contain the current slice.
Decorators
Daiquiri provides a few useful decorators for ComponentResources:
Ensure the user has control before running this resource:
from daiquiri.core import require_control
class ExamplesResource(ComponentResource):
@require_control
def post(self, **kwargs):
pass
Ensure the user is a staff member before running this resource:
from daiquiri.core import require_staff
class ExamplesResource(ComponentResource):
@require_staff
def post(self, **kwargs):
pass
SocketIO
As well as creating REST resources a component can also send and recieve websocket requests via SocketIO, two convience methods are provided to components.
A namespace is created automatically for each component with the same prefix as the _base_url.
emit allows components to emit a SocketIO message:
on allows a component to respond to a SocketIO message (much less frequent use case)
The namespace can be overriden via:
Actors
Components can create and launch Actors which are defined in the implementors python module specified in the component's config file. The actors of a component are in a subdirectory whose name is the lowercase class name. So for Example this would be implementors/example.
Required actors for a component should be defined in self._actors:
This tells the component that two actors are available to it, they must then be mapped to the relevant implementor files and classes via the component config file actors key:
This allows flexibility in mapping what the component requires and where the files live, generally the key and value will be the same. Daiquiri will resolve the value to the example subdirectory in the implementors module i.e. implementors.example.<value>. Filenames should be lowercase and classes should start with an uppercase letter.
Module Resolution
The loader expects the daiquiri classname in lower case. It will resolve to, for example:
- actor1 -> daiquiri.implementors.example.actor1.Actor1
- actor2 -> daiquiri.implementors.example.actor2.Actor2
- actor2 -> daiquiri_id00.implementors.example.actor2.Actor2
Once registered actors can be launched as follows:
actid = self.actor(
"actor1",
spawn=True,
success=self._actor_finished,
error=self._actor_failed,
actargs={
...
},
)
The return value is a unique hash for this actor.
The following keywords are accepted:
| Key | Type | Default | Description |
|---|---|---|---|
| spawn | bool | false | Whether to launch this actor in its own greenlet, otherwise it will be placed into the queue |
| enqueue | bool | false | Whether to put the actor in the queue, if spawn and enqueue are false the actor is placed at the top of the queue |
| start | func | None | A callback when the actor starts |
| success | func | None | A callback if the actor succeeds |
| error | func | None | A callback if the actor fails |
| remove | func | None | A callback if the actor is removed from the queue |
| actargs | dict | {} | A dict of parameters passed to the actor |
The callbacks have the following signatures:
def start(self, actid, actor):
...
def success(self, actid, return_value, actor):
...
def error(self, actid, exception, actor):
...
def remove(self, actid, actor):
...
Actor Resources
An @actor decorator is provided that can automap a ComponentResource to launch an actor.
The simplest case will just launch an actor with the keywords from the request. User input will be validated according the the Actor Schema:
class ActorResrouce(ComponentResource):
@actor("actor1", enqueue=True, [...**actargs])
def actor(self, **kwargs):
pass
The second case with the preprocess keyword allows for preprocessing of the user input by a secondary preprocess function. This can add extra parameters and pass these into the actor. The function should return the updated **kwargs.
A more complex example:
class MosaicResource(ComponentResource):
@require_control
@actor("mosaic", enqueue=False, preprocess=True)
def post(self, **kwargs):
"""Create a tiled mosaic actor"""
pass
def preprocess(self, **kwargs):
kwargs["absol"] = self._parent.get_absolute_fp(
{"x": kwargs["x1"], "y": kwargs["y1"]},
{"x": kwargs["x2"], "y": kwargs["y2"]},
)
sample = self._metadata.get_samples(sampleid=kwargs["sampleid"])
if not sample:
raise AttributeError(f"No such sample {kwargs['sampleid']}")
sessionid = g.blsession.get("sessionid")
def save_image(x, y):
return self._parent.save_image(
sessionid=sessionid,
sampleid=kwargs["sampleid"],
file_prefix=f"mosaic_{x}_{y}_",
)
kwargs["sessionid"] = sessionid
kwargs["save"] = save_image
return kwargs
Actor decorator args:
| Keyword | Description |
|---|---|
| enqueue | Place the actor in the queue |
| spawn | Spawn the actor immediately in a new greenlet |
| synchronous | Spawn the actor immediately in a new greenlet, and await its termination, returning its value in result |
Accessing Components
Components can request other components via:
i.e. a component could get the scans component to retrive scan data:
scans = self.get_component("scans")
scalars = scans.get_scan_data(scanid=dc["datacollectionnumber"], per_page=1e10)
This should be used sparingly as it couples components together.
Flask request context
The flask request context g is populated with some useful things during a request (not valid outside of a http request!):
g.user
Contains an instance of [daiquiri.core.metadata.user][] for the current user. This allows to check for example if the user is a staff member:
Or ask which permissions the user has:
g.blsession
Contains a dict of the currently selected session from the metadata handler: