r17 - 20 Jun 2013 - 10:49:04 - FredericBastienYou are here: TWiki >  Public Web  > BramsUserGuide

Brams Users Guide

This page explain what brams user need to know about our computing environment to use the brams cluster.

If you have question, you can send them to Frédéric Bastien <nouiz@nouiz.org> and Nathanael Couture-Lecaude <n.couture-lecaude@umontreal.ca>.

Connection

To connect, you must connect by ssh to our access point and move to other computer after that.

Our access server are: {elisa1,elisa2,frontal,frontal07}.iro.umontreal.ca

elisa1 and elisa2 are the access point for our lab. frontal for the department.

frontal07 is special in that it don't block the connection that don't have a good reverse DNS. Some ISP don't implement that correctly.

please use frontal07 only if the 3 others fail.

NEVER RUN JOBS ON THOSE COMPUTER!

Computer Usage

screen: You SHOULD NOT use the "screen" command directly. You must use "pkscreen" instead. This will create a detached screen session, to which you should reattach with "screen -r". If you don't do this, it causes trouble for you and the admins with the security system. You MUST ALWAYS user pkscreen. There is a few use case where you could use only screen, but DON'T DO THIS. If you prefer tmux, you can run it inside the screen. Otherwise there will be the same problem like with the normal screen without using pkscreen.

Computer usage

  • Linux introduction? for those that need it.
  • Log out of your computer if you won't be coming in the next day, or when you are leaving for the weekend. This helps to keep the computers stable(...) AND allows the computer to be used by other students who might be in the lab at different times than you. Also, if you can log out every day, that is even better.

  • screen: You SHOULD NOT use the "screen" command directly. You must use "pkscreen" instead. This will create a detached screen session, to which you should reattach with "screen -r". If you don't do this, it causes trouble for you and the admins with the security system. You MUST ALWAYS user pkscreen. There is a few use case where you could use only screen, but DON'T DO THIS.
    • If you prefer tmux, you can run it inside the screen. Otherwise there will be the same problem like with the normal screen without using pkscreen.

  • ~/.bashrc: You MUST load the configuration of our lab. This give you a much better Linux environment. To do so, put this line into your ~/.bashrc file:
 
       if [ -e "/opt/lisa/os/.local.bashrc" ];then source /opt/lisa/os/.local.bashrc; else source /data/lisa/data/local_export/.local.bashrc; fi

  • File system:
    • You MUST put your experiment scripts and their result under /data/lisa/exp/$USER. This prevents everybody's home directories going down if there is a problem with your jobs or there are too many jobs running at the same time. There are some exceptions, if Fred tells you something about this.
    • You MUST put your final datasets under /data/lisa/data.
    • You MUST use /data/lisatmp or /data/lisatmp2 for everything that you don't want under backup. Their is no backup of that directory
      • When you build a dataset, create it under /data/lisatmp[2] as this is big file that you will probably change frequently at the beginning. Put the final dataset under /data/lisa/data
    • There is a soft quota of 100G for the home directory. This mean that if you bust it, you have 7 days to fix the problem before you can't write anymore.
    • We backup completly each changed files. Even if you just append to the file, we backup it completely.
    • Don't do chmod on files that don't need there permission changed. chmod modify each file even if they don't need to modify the permission. This make the backup system to re-backup everything even if it didn't changed the permission!
      • To work around that, you can use find to modify only the file you want. Example to give the group read permission on file and read/execute on directory:
        • find ! -perm -g+rX -exec chmod g+rX \{} \;
  • Good use: Most stuff is not regularly checked for good use by all students. We just ask that people try in good faith to use them correctly and try to share the resources of the lab with all of the people in the lab. Mistakes happen, learn from them and try to update this page to warn new/other lab members about them.

  • SSH: It is a good habit to have the mininum number of hops to the final destination of your ssh connection. Having many of them makes your connection less stable. It requests more resources for the whole computer infrastructure and makes diagnostic of problems harder. So try to avoid useless intermediate ssh connections: you should either log directly at a workstation (direct ssh) or connect from the outside to the login node (elisa{1,2}) and then ssh to the destination computer. If elisa1 and elisa2 are not working, you can try frontal and frontal07, but this should be temporary and don't make screen on those computer. Use the frontal* computer only to connect and go to another computer after that.

  • NX server We have an NX server with very low support. A user told that using "ssh -XY" when connecting back to our lab computers (instead of simply ssh -X) solve problems.
    • Alternatively, you can add "ForwardX11Trusted=yes" to your .ssh/config file.

  • ncdu a command line tool that help you clean up your files.

Cluster documentation

Our cluster documentation is at:

http://www.iro.umontreal.ca/~lisa/twiki/bin/view.cgi/Public/LisaCluster

The documentation is not very clear, so don't hesitate to contact us if you have problem.

Matlab

  • All of you and members of our lab should use a maximum of 6 matlab license together.
  • For now, try to coordinate this between you
  • I will try to make some more easy by the end of the year, but I can't warranty it.
  • If you need more then the 6 license, we will need our admins to find a way to use your license here.
    • This can take time, so you must plan it.
  • OLD STUFF not done with Fedora 14. If you need it, I can redo this:
    • to be able to use spm8, you must add the path into matlab `/opt/lisa/os/matlab_toolkit/spm8`
      • as a command line: `addpath /opt/lisa/os/matlab_toolkit/spm8`
      • in the gui: File > Set Path > Add Folder
    • to be able to use Field Trip, add to your matlab path : /opt/lisa/os/matlab_toolkit/fieldtrip/

Additional point not in the documentation

  • To submit jobs to our cluster, you must do it from the zappa8 computer.
  • To run tests jobs by ssh, you must do it on the maggie46 computer. DON'T USE ZAPPA8 as this cause instability and when this happen, it can kill all running jobs...
  • To submit jobs, you must use the command jobdispatch as described in the [[http://www.iro.umontreal.ca/~lisa/twiki/bin/view.cgi/Public/LisaCluster][doc].
  • The jobs scheduler that we use is condor. jobdispatch only submit the jobs to condor and make many boiller plate work.
  • Computers:
    • To submit only to brams computer, you must add the parameter `--machine=brams0{1,2,3,4,5,6,7,a,b,c,d,e,f,g}.iro.umontreal.ca` to the jobdispatch command
  • Filesystem: All files(dataset, script, executable, result) for an experiments must be under on of those folders `/data/gamme/{avec,sans}-bkp/exp/$USER`.
    • If you don't do that, you will have problem with your home fileserver. The home fileserver is under NFS4 with kerberos(for security) but their is problem with Fedora 14 NFS client and the NFS server for jobs in that configuration.
    • `/data/gamme/{avec,sans}-bkp/exp/$USER` is on another computer that is NFS4 only. So we bypass some of NFS/kerberos interaction bug.
    • Also /data/gamme/{avec,sans}-bkp/exp is much faster
  • You can submit all your jobs at the same time but allow only X to run at the same time(for matlab licences, to don't overload the fileserver if that is a bottleneck in your application, to be nice...)
  • For this add the parameter `--nb_proc=X` to `jobdispatch`
  • This send 1 jobs that will act as an hyper-manager that will submit only X jobs to condor(the jobs scheduler that we use)
  • To see the currently submited jobs by this hyper-manager execute `condor_q -dag`.

-- FredericBastien - 07 Dec 2010

Edit | WYSIWYG | Attach | Printable | Raw View | Backlinks: Web, All Webs | History: r17 < r16 < r15 < r14 < r13 | More topic actions
Public.BramsUserGuide moved from Public.BramUsers on 07 Dec 2010 - 15:18 by FredericBastien - put it back
 
Home
This site is powered by the TWiki collaboration platformCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback