Therefore, the Ceiba web service needs to handle two kinds of requests by the client: read-only queries and mutations on the datasets. These “queries” and “mutations” can be easily describe with GraphQL.
In a nutshell, GraphQL defines a contract (known as a schema) between the actions that a client can perform with the web service and the possible outcomes of those actions. More formally, GraphQL is a query language that allows you to specify an application Programming interface (API) using different programming languages. If you have previous experience with RESTful API have a look at a comparison between GraphQL and REST.
But how does GraphQL work? First, you need to define a schema using the GraphQL schema language. The following code snippet defines a schema to query a job using its status,
Schema definition for job query
The Query** schema specifies that in order to request some jobs* you need to provide a Status argument, where Status can be one of four possibilities: AVAILABLE, DONE, FAILED and *RUNNING. *The exclamation mark (!) indicates that the argument cannot be Null (a.k.a None in Python).
The following Mutation schema defines the required arguments to update a given job status.
Schema definitation for Job status mutation
The updateJob action specifies that you must provide an id and a new_status in order to be able to update a job. You will receive a Reply specifying whether the update action has succeeded.
Have a look at the Ceiba queries and mutations schemas. They are slightly more complex than the aforementioned schemas but follow the same rationale as the previous examples. You can also have a look at the official introduction to GraphQL.
We have just defined the schemas that specify the actions that we want to perform. We still need to implement the actions and for doing so, we need a GraphQL engine: a library that takes the schemas together with the code that implements the actions and generates an API.
We have chosen the Tartiflette GraphQL engine to implement our web service mostly because it is easy to use and open source. The following snippet shows a possible implementation for querying jobs based on their status using Tartiflette.
the Resolver decorator indicates that the resolver_query_jobs function corresponds to the implementation of the query jobs schema. The function takes 4 arguments of which I only use args and ctx(You can refer to Tartiflette for further details). ***args ***contains the arguments given by the client code, while ***ctx ***contains the context for running the current function, for example the handler to access the database that is called mongodb in this code snippet.
Notice that the definition of the aforementioned function starts with the async keyword. Asyncio is a popular built-in Python library to write concurrent code. It is extensively used to write high performance web services.
In the Ceiba web service implementation of the queries and mutations, there are definitions for all the Python functions that perform the actions specified in the GraphQL schemas. For each query and mutation, there is a corresponding function.
The database
We need a database not only for storing the interesting data but also to store the jobs metadata, like what jobs are available. For the Ceiba web service we use MongoDB.
My personal opinion is that a NoSQL database like MongoDB gives a significant advantage over traditional SQL databases on research projects where up-front design of the schemas to store data is unfeasible. The research priorities can change as the project evolves and having dynamic schemas to store the data makes the researchers’ lives easier.
Putting all together
Photo by frank mckenna on UnsplashDocker containers are the perfect way to ship our web service. We just need to write a [Dockerfile](https://github.com/nlesc-nano/insilico-server/blob/master/
Dockerfile) with the recipe to install and start the service together with the mongo container.
If you want to deploy the Ceiba web service to a remote server you need to follow these steps:
- Install Ansible in your computer.
- Clone the Ceiba repo and go to the provisioning folder.
- Edit the inventory file with the address of the server(s) where you want to install the runner.
- Edit the playbook file with the
remote_username for the remote servers. - Make sure that you can ssh to your server(s).
- Install the runner with the following command:
ansible-playbook -i inventory playbook.ymlThe Ceiba server should be up and running!
The pesky details
You certainly do not want to keep your web service open, so people can remove your data. You want that users are authenticated before using your service, but you also do not want to manage all the security on your own. Getting authentication right using something like OAuth2 is tricky and it needs at least an entire post on its own.
Also, you need to host your web service somewhere and hosting costs money. It is simply not viable that you host your service in your computer, it is not safe and it takes too much time to maintain. Fortunately for researchers, there are institutions like SURF that can help you to host a web service for research purposes.
Acknowledgement
Creating the Ceiba web service would not be possible without Stefan Verhoeven advice and the computational resources provided by SURF.
I will also to thank Jens Wehner, Nicolas Renaud, Johan Hidding, Pablo Lopez-Tarifa and Victor Azizi for their feedback and support.
Specially thanks to [Patrick Bos,] Tom Bakker for their feedback.
