Irvine Lab Code Documentation Home Page

Static Packages:

Personal Pages (packages in development)

How to use ILPM

Instructions for how to use ILPM are here, and similar instructions are found on the main sites of other packages.

How to use the Git repo

The git repository lives on the labshared3 volume of ImmenseIrvine. Administrators with root privileges can create new repositories (such as personal development repos for new members) on the server via 

$ ssh root@immenseirvine.uchicago.edu
$ cd git
$ ./make_repo.sh repo_name

This executes a shell script that makes a new repository. If you do not know the administrator password for Immense, and hence are unable to log in as root, then please ask an administrator to do this for you (Noah or Bob, for instance).

If you are unable to cd git above, this probably means that the symbolic link called git has been erased in a Synology Diskstation update. If this is case, log in as root and run ln -s /volume3/git git.

Instructions for how to use the git repo are here.

Using Midway

Midway is the cluster that we use for large computational jobs. The main Midway documentation is quite excellent, and that should be used as a primary resource. Nevertheless, some basic definitions remain a bit nebulous throughout the documentation. Follow this link to a page where we add bits of information and definitions that are useful in using Miday.

How to add to IrvineLabCodes

You have some code. You think its pretty cool and want to share it with the group. People start asking you for parts of the code because it is of general use. What do you do?

Firstly, everyone in the group gets a personal repository for their code (and code only, no data, images, etc allowed in the repos). While your package is changing and maturing, keep it in your personal repo.

Now let's say you have a self contained package that is designed for, say, tracking curves in 3D or creating networks or whatnot. You have two options: (1) distill it into a well-documented, single module to add to ILPM, or (2) make it available as a standalone package in the Irvine Lab Code Documentation here.

If you have a well-written, well-documented, broadly useful, static package with minimal additional dependencies, then:

  1. If the package can be distilled into a module that is complementary to ILPM and interfaces well with ILPM's other functionality, add it directly to ILPM.
  2. If the package is necessarily larger (with many modules), is more specialized, or has many non-standard dependencies, you can package it as an egg, create that package as its own repo on the server (now separate from your personal repo), and include the documentation here, next to (and separate from) ILPM.
  3. If the package doesn't meet the above criteria, keep that code in your own repo where you keep your code version-controlled. Then store your docs here, so you and others can search through them and refer to your example usage code. For an example on how one can do this, see the LEPM package.

For (1), read Getting started with documentation and then read the section How to add to ILPM.

For (2) or (3), read Getting started with documentation and then read the section How to create your package.

Getting started with documentation

Here are some helpful tips for how to add code to either ILPM or add a separate package, such as a personal package or a shared package like a GPE python modules package. For any additions (either of the above), you will need to run Sphinx to automatically generate code. After these initial steps, in later sections we'll touch on how to specifically add a new package or to modify an existing one (like ILPM).

First download Sphinx: To begin, if don’t have pip, download get-pip.py binary from http://pip.readthedocs.org/en/stable/installing/ then 

$ sudo python get-pip.py

Run: 

$ pip install -U Sphinx

You may have to run it with sudo, depending on your permissions: 

$ sudo -H pip install -U Sphinx

Also install numpydoc, which lets you use a nicer commenting format in your code (as described here for example: https://codeandchaos.wordpress.com/2012/08/09/sphinx-and-numpydoc/): 

$ sudo easy_install numpydoc

Now for an example: a project called helloworld. Pull it from the Git repo (see the 'How to use git repo' page if this sounds daunting). It is located at ssh://YOURNAME@immenseirvine.uchicago.edu/git/helloworld. helloworld is an example of a package. Any package you want to add to ILCD (rather than just as a module in ILPM) should have the file and directory structure of helloworld. It also has helloworld/doc/conf.py and helloworld/doc/Makefile which you can use for creating your own repo's Sphinx-generated documentation (after some minor edits to the conf.py, as we will discuss later in the How to create your IrvineLab package section). Let's develop the helloworld package. This means that anywhere in your computer, you can enter a python environment and type import helloworld, and it will work. In other words, the helloworld package is globally 'installed', but it's actually 'installed' in a way which lets you make changes to the modules that will instantaneously be recognized everywhere. That's what it means to 'develop' a package rather than installing it: the modules are symbolically linked in your packages directory, not hard-copied. It is going to be necessary for our code documentation procedure for you to develop (or install, if you absolutely must) your package, unless you are just adding a (set of) script(s) to ILPS. Ok, so let's develop: 

$ sudo python setup.py develop

You can now check that you can import helloworld from anywhere in your operating system.

Now for the documentation itself: let's build the html pages using Sphinx. There is a conf.py in the package already, which control how the html will look and how it is named. There is also a Makefile which knows about where the information that goes into the html is located. (Note that you will have to set up a similar structure in your own package in order for a copy of this Makefile to work in the context of your own package.) There are two .rst files in which tell Sphinx what content to use to populate the HTML: index.rst Go to path_to_helloworld/helloworld/doc/ and enter 

$ make html

You should see a _build directory appear inside path_to_helloworld/helloworld/doc/. Open up the index.html file in a browser. It should be identical to this site. Congratulations! You made some documentation!

How to add to ILPM

If you have not already checked out Irvine Lab Python Modules (ILPM), do this through SmartGit (recommended, standalone GUI software) or Git. If you use SmartGit (recommended), follow the 'How to use git repo' page. Otherwise you can do: 

$ git clone ssh://YOURNAME@immenseirvine.uchicago.edu/git/ilpm
$ cd ilpm
$ sudo python setup.py develop

Before making the html files for ILPM, in order for it to work, you need to be able to use the ILPM codes (i.e. they must function). This means that pyopencl needs to be installed on the computer. If it is not installed, run: 

$ sudo pip install Mako
$ git clone http://git.tiker.net/trees/pyopencl.git
$ cd pyopencl
$ git submodule init
$ git submodule update

$ cd pyopencl-VERSION #2012.1 if you're not there already
$ python configure.py
$ make
$ sudo make install

Make the html files for ILPM on your local machine: 

$ cd $HOME/ilpm/doc/
$ make html

The html pages will then be in <ilpm git repo>/doc/_build/html/.

Once your module is ready to be added to ILPM (you're no longer editing it, and it has been tested throughly), you can add it to ILPM as follows. First, pull ILPM to make sure your copy is completely up to date (except for your new addition). Then move the module to ilpm/cod/yourmodule.py and its rst file to ilpm/cod/yourmodule_rst_file.rst. Then run 

$ build html

from the doc folder in terminal. Immediately update the git repo with SmartGit.

If you are unsure how to write your rst file for your module, follow the examples of other modules in ILPM and helloworld. Be sure to write an Example Usage section, as this is the primary way that people will understand the gist of your module.

How to create your IrvineLab package

If you have a module that is somewhat specialized but still of broad use or has many non-standard dependancies, package it as an egg, create that package as its own repo on the server, and include the documentation here, next to (and separate from) ILPM.

You can make your documentation page using ilpm as a template. Make yourself a repository directory, projectname/, then in a documentation subdirectory add a dir called projectname/doc/ and copy conf.py and Makefile from the ILPM repo (don't move/delete them from ILPM!). Also copy the entire _static subdirectory to your project directory (projectname/_static/). The location of your codes in this case would be projectname/projectname/, and your package could/should be packaged as an egg and have __init__.py files in the right places (see the ilpm directory structure for an example).

Now you'll edit your conf.py configuration file. Make sure there is a line near the beginning of conf.py like 

sys.path.append(os.path.abspath("../project"))

before the ‘General configuration’ section.

You might define your extensions like: 

extensions = [
'sphinx.ext.autodoc',
'sphinx.ext.mathjax',
'sphinx.ext.viewcode',
'matplotlib.sphinxext.mathmpl',
'matplotlib.sphinxext.only_directives',
'matplotlib.sphinxext.plot_directive',
'sphinx.ext.autodoc’,
'numpydoc',
'sphinx.ext.autosummary',
        ]

Make sure the html_theme is ‘classic’ and and make sure the html_logo is ’irvine_trans.png’, which you have already grabbed from the ILPM directory (in projectname/_static/).

If you want more Sphinx extensions, or if your numpydoc isn’t working for some reason, you can grab some from the sphinxext directory here: https://github.com/matplotlib/sampledoc

For each module in a project, add an rst file in the same dir as conf.py. Edit index.rst to link to each module's page by listing that rst file in index.rst.

Example plots can be made by saving demo_name.py files and using the plot directive in the rst file: 

.. plot:: ./path_to/demo.py
    :include-source:

where demo.py opens plots (and does not close them). Leaving the plots open signals to Sphinx that it should include the plot in the html. Blocks of code (that may or may not actually work/run) can be included: 

.. code-block:: python
    code blah blah blah

or 

.. code-block:: other_language
    code in other_language blah blah blah

Now that you have the ingredients in place, create the html by running 

user@machine:path_to_project/projectname/doc$ make html

That should be enough to get you started.

As always, be as selective as possible when syncing with the git repo, and avoid extra files, especially images, whenever possible. Do not include html in the git repo, or anything from a _build directory created by Sphinx. Only include the tools to make that html, including the conf.py and Makefile (which are both in projectname/doc/

How to add a script to the IrvineLab Python Scripts

Scripts are python codes that have no helpful features of a module. They are meant to be run, not imported. These are useful for data processing pipelines, etc. To add a script to the IrvineLab Code Documentation, follow the same instructions as for adding to ILPM, but instead add your scripts in a directory inside the ILPS repo, located at ssh://YOURNAME@immenseirvine.uchicago.edu/git/ilps.