HOWTO

Contents:

• Development Checklist
  • 0 - Copy mg-process-test from GitHub
  • 1 - Setup Your Python environment
  • 2 - Create a Tool
  • 3 - Create a Test to run the Tool
  • 4 - Create a Pipeline
  • 5 - VRE JSON Configuration
  • 6 - Installation Documentation
  • 7 - COMPSs testing
  • 8 - Hook up your repository for continuous integration
  • 9 - Congratulations
• HOWTO - Tools
  • Repository Structure
  • Basic Tool
  • pyCOMPSs within the Tool
  • Practical Example
  • Troubleshooting Common Issues
• HOWTO - Pipelines
  • Example Pipeline
  • Code Walk Through
• HOWTO - Documentation
  • Building the Documentation
  • Updating the documentation
• HOWTO - Licensing
  • Implementing the Apache 2.0 license
  • Benefits for developers
• HOWTO - Configuration Files
  • Tool Description
  • Test Configuration Files
• HOWTO - Logging
  • Levels of Logging
  • Using Logging
• HOWTO - Testing Your Code
  • Running the Code
  • Testing a Tools and Pipelines
  • Automated Testing
  • Running in COMPSs

The following is a walk through of developing a tool and pipeline wrapper to include new functionality within the MuG VRE. There are several stages covering the Tool development, using the tool within a pipeline and defining the configuration files required so that the final product can be smoothly integrated into the MuG VRE.

Common Coding Standards

When it comes to developing the code all the code should stick to a common standard. This has been defined within the Coding Standards documentation as well as how to set up the licenses correctly so that your package can be integrated.

Adding a new function

All of the examples in the following sections describe code that has been incorporated into a functional pipeline and tool within a demonstration VRE Tool that is ready for deployment within the VRE. The code can be found in GitHub repository mg-process-test.

GitHub mg-process-test

In the test process there are example pipelines, tools, documentation, setup scripts unit tests and config files. This repository can be forked and used as the base for developing new pipelines and tools.

The following documents will help guide with the creation of all the components required for creating a tool ready to be integrated into the VRE. To help with the development and Development Checklist has been created to provide a generic guide and checklist to help make sure that nothing has been forgotten.

Wrapping a Tool

This section guides you through how to wrap an external tool, or create a tool that utilises the pyCOMPSs framework and should be capable of running within the MuG VRE environment.

Creating a Pipeline

Once you have created a tool you can now incorporate one or multiple tools into a pipeline. This will handle the passing of variables from the VRE to the tool and the tracking of outputs ready for handing back to the VRE. This document will also help in creating test input metadata and file location JSON files that are required to run the pipeline.

Documentation

This provides a overview of the documentation requirements as described by the MuG Coding Standards.

Logging

Takes you through adding logging to your pipelines and tools to return messages to the user via the MuG VRE.

Testing Your Code

A important part of making sure that a pipeline or tool is ready for integration is ensuring that the code has been tested. This covers testing the code is functional and that it is capable of running within the infrastructure used by the VRE.

Licensung Your Code

The Apache 2.0 license is required for pipelines and tools to to be integrated into the MuG VRE.

VRE Configuration

This takes you through creating JSON configuration files for your tool. This should define all the inputs, outputs and any arguments that are required by the pipelines and tools.

Integrating a new tool into the VRE

The next step is the integration of the pipeline/tool into the MuG VRE. The Configuration should provide guide to the initial JSON files. The full JSON specification is located in this GoogleDoc. The details the requirements for correctly creating the Tool description JSON file and the requirements for parameters needed for an application.