Posts

• Aug 12, 2016

  xarray and terrabyte datasets

  As models get to higher and higher resolutions researchers have an ever increasing amount of output data to sift through. The 1/10th degree ocean model currently being used in the Centre spits out 70 GB of data per month for temperature alone.

• May 20, 2016

  Python 3 at NCI?

  This week I’ve been thinking about how we’ll manage Python packages for the ACCESS community at NCI to add better support for Python 3. I’ve started adding Python 3 support to my own scripts (using travis.org to check they actually work), and it’s not been that bad, and we’re starting to get users ask on the helpdesk for Python 3 versions of the climate-focused packages we support.

• Oct 30, 2015

  Distributing Python Libraries, Part 2

  In my last post I talked about how to set up a new python library by creating a new source repository and the setup.py script that pip uses to install it. In this post I’ll talk about setting up testing for the library, and automating code quality checks using services like Travis-ci and Landscape.

• Oct 23, 2015

  Distributing Python Libraries

  Recently I created a Python library called ARCCSSive, which allows researchers to search through the CMIP5 archives mirrored at NCI. I also wanted to use this as an opportunity to investigate how to make a high-quality library that can easily be used by others. This includes putting the library into public source control, setting up documentation, unit tests & coding standards, and publishing the library to the Python Package Index. I also wanted all of this to be as automated as possible - so that I could create a new release of the library with a single command and know it’s tested, documented and available to others.

• Jun 19, 2015

  Custom Types in Fortran

  Types like INTEGER and LOGICAL are used by the compiler to make sure you’re passing the correct arguments to functions. This is really helpful when functions have lots of arguments, as it lets you know if you’ve missed one, or entered them in the wrong order. You can create your own types to bundle together related variables, or to provide more granularity to argument checking.

• Jun 5, 2015

  Python Environment Modules

  Raijin, NCI’s supercomputer, like many HPC environments uses Environment Modules to provide access to programs and libraries. Using Modules allows different users on the supercomputer to access different versions of compilers and libraries without having them conflict or messing about with PATH environment variables. We’ve been using environment modules in the ACCESS area for a while now to provide access to Climate focused tools and libraries, and have built up a decent stash of Python libraries.

• Jun 4, 2015

  Backing up the CMS cloud

  The CMS cloud is starting to store useful data, so I’ve been looking at ways to set up backups. I’ve been using Amanda on my workstation for a couple years now, backing up to an external hard drive, and I wanted to set up something similar.

subscribe via RSS