Chapter 3 Server

3.1 Login

Once the server administrator ( contact: researchit@iastate.edu ) setup your account on server in Dr. Li’s group, you can ssh to the server.

Open your terminal (Linux/Mac) or Git-Bash (Windows) to SSH to pronto.las.iastate.edu and log in with your NetID and university password.

ssh zhy@pronto.las.iastate.edu

3.2 Directories description

home ~ directory can be accessed only by yourself. It has very small spaces, so only put some some small files and configuration files here.

work /work/LAS/liganwu-lab directory can be accessed by the whole group. It has larger spaces. Put all your data and scripts needed in a job here. /work is not backed up, so copy all important data to /lss for backing up in case the lost of the data.

lss /lss/research/liganwu-lab is for long-term file storage. Dr. Li has purchased 5T so far. Make sure to only put important files here.

3.3 Data transfer

Use data transfer node : prontodtn.las.iastate.edu

You can use both scp or FTP (Filezilla) methods, GUI FTP is easier.

Filezilla site manager settings:

ftp setting

ftp setting

command line scp example


# to upload  all the .gz file in the current dir of local computer to  server  /work/LAS/liganwu-lab/ying/run_meta_pl1/2019-05-28

scp *.gz  zhy@prontodtn.las.iastate.edu:/work/LAS/liganwu-lab/ying/run_meta_pl1/2019-05-28

# to download from the server "/work/LAS/liganwu-lab/ying/run_meta_pl1/2019-05-28/kraken_kaiju_combined"  to  current dir of local computer: 

scp -r  zhy@prontodtn.las.iastate.edu:/work/LAS/liganwu-lab/ying/run_meta_pl1/2019-05-28/kraken_kaiju_combined  .

3.4 Run jobs

Make sure you are on pronto, ssh zhy@pronto.las.iastate.edu. Once you’re on pronto, you’ll need to use the slurm job scheduler to run a job (either interactively or in batch mode).

For information on how to use the slurm job scheduler:

https://researchit.las.iastate.edu/pronto

https://researchit.las.iastate.edu/slurm-basics

interactively: Interactive jobs give you a shell prompt to interact with, much like you did when directly ssh’ing to a server.

Interactive session example

To get an interactive session for an hour on one node with 8 cpus. (It is useful when you need to install some software on server.)

srun --time=01:00:00 --nodes=1 --cpus-per-task=8 --pty /usr/bin/bash

batch mode: Batch jobs allow you to submit a script to the queue, and it is run as soon as possible (independent of you being logged in).

most of the jobs can be done using batch mode.

3.5 basic slurm commands

(https://slurm.schedmd.com/quickstart.html)

  • sbatch is used to submit a job script for later execution. The script will typically contain one or more srun commands to launch parallel tasks.
sbatch myjobscript
  • scancel cancel a job
scancel jobid
  • squeue reports the state of jobs or job steps. It has a wide variety of filtering, sorting, and formatting options. By default, it reports the running jobs in priority order and then the pending jobs in priority order.
squeue
squeue -u zhy
  • sacct is used to report job or job step accounting information about active or completed jobs.

3.6 Cloud server such as AWS

  • later