Set up your Google Project
In order to provision a cluster, you will first need to obtain an account on the Google Cloud Platform, if you don’t already have one. You can find information here: https://cloud.google.com/.
Once you have an account, navigate to the Google Cloud Console at https://console.cloud.google.com and create a project. See https://cloud.google.com/compute/docs/projects for instructions about how to create and manage your projects.
Once you have created a new project, make sure you enable Compute Engine in this project. You need to enable Compute Engine project by project. To check that this has been done, try creating at least one compute instance. Pick the cheapest type of instance, this is just needed
to trigger compute engine configuration. After the instance has started, you can delete it because you have now confirmed that your project is enabled with Compute Engine.
To obtain credentials, open the Google console, select the Project you wish to use, and locate the API Manager tab. You will see Overview and Credentials. Click on Credentials.
You may find it easier to navigate to https://console.cloud.google.com/apis/credentials and then select the Project you wish to use.
In the Credential tool, click on the “Create credentials” button and then select OAuth Client ID. Complete the information requested and save. Once credentials are generated, you can obtain the following information:
- Client ID
- Client secret
- Project ID (this is the name of the project to use when creating instances, note that the id is all lowercase and shown to the right when you click on the project name in the console and look at a list of projects)
Configuring the credentials in the project
Open MPS > Preferences… > Other Settings > GoogleCloud and then enter the project credentials in the form:
The credentials are NOT stored in the project, therefore they will not be committed if your project is under source control.
Provisioning a cluster
At this point, you have everything you need to create a new cluster in your project. This can be done in NextflowWorbench by importing the org.campagnelab.compute.cloud in the current model.
Now you can create an instance of “Manage Cloud Cluster” by right clicking on the model and select New > o.campagnelab.cloud > Manage Cloud Cluster
This will create a user interface that allows to fully manage a cloud cluster on the Google Platform. To start and activate the cluster, just click on the Create button. Almost immediately, you will be prompted to enter a verification code. This means that you need to approve the cluster creation from your Google account.
At the end of the creation process, you will see:
Configuration for Users
To allow users to submit workflows in the new cluster, you have to:
1) register the user’s public SSH keys in the cluster
The public SSH key of the user must be registered in the cluster to allow NextflowWorkbench to establish a network connection with the cluster. The Grant Access button on the user interface allows you to do that with simply one click. Just paste the user key where you see <paste public ssh keys here> and click on the button. This way, the key will be transferred to the cluster and the user will be able to connect to it for submitting his/her workflows.
Do note that more than one user key can be added at the same time.
2) give the users the minimal configuration they need to access to the cluster
The List Nodes button shows the information needed to configure workflow submission in the workbench.
Users aiming to submit workflows in the newly created cluster must be provided with:
- the IP address of the front-end
- the username to log in the front-end node (usually ubuntu in the default configuration)
- a folder under the user home that you want to use as working directory for the workflows.