Creating Packages¶

Feedstocks for Azure Pipelines may be generated based on the following sources (listed in the order of preference):

conda-forge feedstocks;
PyPI packages;
package recipes from lightsource2-recipes.

For a new package, the recipes (in the form of meta.yaml file) can be created manually from a generic template for meta.yaml file and specific package requirements.

Directory structure¶

The following directory structure for the packaging project is recommended:

~/src/nsls-ii-forge/ – root directory in which packages will be placed. This is only a recommended name. In practice, root directory may have arbitrary name and placed in any convenient spot.

~/src/nsls-ii-forge/event-model-feedstock – the directory with cloned event-model-feedstock repository.

For each package we will create a temporary directory that may be deleted after a feedstock is initialized and the feedstock directory that will contain the feedstock repository:

~/src/nsls-ii-forge/<package_name> – a temporary directory with the package’s recipe. The directory is created by conda skeleton or manually. The directory name has to match the package name, since this is the directory where conda-smithy will be looking for the recipe file meta.yaml (for example, recipes for sixtools package must be placed in ~/src/nsls-ii-forge/sixtools directory).

~/src/nsls-ii-forge/<package-name>-feedstock – a directory that contains the feedstock repository, which is uploaded to the corresponding repository in the nsls-ii-forge GitHub organization. The directory is created by conda-smithy as a part of feedstock initialization.

Set up the development environment¶

Install “conda-smithy”¶

Create a new conda environment (let’s name it smithy):

$ conda create -n smithy -c conda-forge conda-smithy "python>=3"
$ conda activate smithy

Install Keybase¶

Use the installation instructions to download and install Keybase. Start Keybase and log into the system (DAMA group). Check if tokens are downloaded to the directory on your local hard drive (/keybase/team/dama/ on Linux and MacOS systems).

Configure “conda-smithy”¶

Create the directory ~/.conda-smithy (in your home directory).

$ mkdir ~/.conda-smithy

Copy tokens named anaconda.token, azure.token and github.token to ~/.conda-smithy:

$ cp /keybase/team/dama/anaconda.token ~/.conda-smithy/
$ cp /keybase/team/dama/azure.token ~/.conda-smithy/
$ cp /keybase/team/dama/github.token ~/.conda-smithy/

Create a root directory for packages¶

Naming and location of the root working directory is arbitrary. We will place packages in the directory ~/src/nsls-ii-forge/:

$ mkdir -p ~/src/nsls-ii-forge

Clone the “event-model-feedstock” repository from the nsls-ii-forge GitHub org¶

The event-model-feedstock needs to be cloned only once. In the process of creating packages we will be using the conda-forge.yml file located in the ~/src/nsls-ii-forge/event-model-feedstock/ directory and the conda_build_config.yaml file located in the ~/src/nsls-ii-forge/event-model-feedstock/recipe/ directory. If you already have it cloned, pull the latest version of the event-model-feedstock since the repository may be updated.

$ cd ~/src/nsls-ii-forge
$ git clone https://github.com/nsls-ii-forge/event-model-feedstock.git

Generate and edit the `meta.yaml` recipe configuration file¶

There are several ways to create the package recipe:

use the existing meta.yaml recipe file (e.g., from a conda-forge feedstock repository);
generate a recipe from the existing PyPI package using conda skeleton;
manually create a meta.yaml recipe file.

Note

It is important that each of the following commands is run from the correct directory. Since it is easy to lose track of directory changes, all the paths will be specified relative to the $HOME directory.

Generate a recipe from a PyPI package¶

If the package <package-name> is available from PyPI, generate a recipe from the existing package:

$ cd ~/src/nsls-ii-forge
$ conda skeleton pypi <package-name> --noarch

Check if meta.yaml was successfully created in the ~/src/nsls-ii-forge/<package-name> directory.

Edit the meta.yaml file:

Remove all entries from the requirements: host: section except python and pip.

Add requires: and commands: to the test: section (if applicable).

requires:
  - pytest
commands:
  - pytest --pyargs sixtools.tests

Remove the following lines from the about: section:
```
doc_url: ""
dev_url: ""
```
Additional steps:
- Add LICENSE to the license_file field of the about: section.
- Update the home URL in about: section to point to the package’s GitHub repository or a dedicated web site if it exists.
- Remove the extra: block (including the list of maintainers).

An example of the edited meta.yaml for the sixtools package may be found at the sixtools-feedstock repository.

Manually create a recipe or use the existing recipe¶

This is an alternative method of preparing the recipe if the package is not available at PyPI. Create a temporary directory:

$ cd ~/src/nsls-ii-forge
$ mkdir <package-name>
$ cd <package-name>

The recipe may be created based on the meta.yaml file from the respective conda-forge feedstock (and sometimes from the original package repository). If such a file is not available or unusable, find an appropriate sample meta.yaml (from a similar package that was successfully built) and modify it. Copy meta.yaml file to the temporary directory you just created:

$ cp <path-to-meta-yaml-file>/meta.yml .

or download meta.yaml from a known URL:

$ wget https://<url-of-meta-yaml-file>/meta.yaml

Open and edit the meta.yaml file.

SHA256 digest¶

When manually creating a recipe it will be necessary to find or calculate the SHA256 digest of the <package>.tar.gz file. If the package has been released on PyPI then the SHA256 digest of the associated <package>.tar.gz file can be found directly on PyPI without the need to calculate it. Set source: url: in meta.yml to the PyPI download link.

If the package has not been released on PyPI a <package>.tar.gz file may be available from GitHub or a similar online source code repository. In this case download the <package>.tar.gz file and calculate its SHA256 digest. The recommended method is to use openssl. For example,

wget https://github.com/bluesky/bluesky/archive/v1.6.6.tar.gz

openssl sha256 v1.6.6.tar.gz

Use the download link from the source code repository for source: url: in meta.yml.

Note

The <package>.tar.gz file on PyPI will not have the same SHA256 digest as the <package>.tar.gz file downloaded from a source code repository such as GitHub.

Prepare recipe files¶

Collect additional files¶

Copy the conda_build_config.yaml file into your recipe directory:

$ cd ~/src/nsls-ii-forge
$ cp event-model-feedstock/recipe/conda_build_config.yaml <package-name>/

Open and inspect conda_build_config.yaml:

$ cd ~/src/nsls-ii-forge/<package-name>
$ emacs conda_build_config.yaml &

This is the contents of a typical conda_build_config.yaml file:

channel_sources:
- nsls2forge,defaults
channel_targets:
- nsls2forge main
python:
- '3.6'

If you are building a noarch package, then close the file without change. For an architecture-dependent package, a set of Python versions are specified in the meta.yaml file and the following lines should be removed:

python:
- '3.6'

Copy the license file from the original package repository into the recipe directory. The license file is typically named LICENSE (without an extension), but in some projects the name may differ (e.g. COPYRIGHT). The spelling of the license file name should match the name specified in the license_file field of the about: section of the meta.yaml file (see the instructions above). For example, the BSD license used for the Bluesky project may be copied to recipes as follows:

$ cd ~/src/nsls-ii-forge/<package-name>
$ wget https://raw.githubusercontent.com/bluesky/bluesky/master/LICENSE

Now the content of the recipe directory ~/src/nsns-ii-forge/<package-name> should look similar to this:

drwxr-xr-x 2 user user 4096 Sep 13 12:45 .
drwxr-xr-x 3 user user 4096 Sep 13 12:10 ..
-rw-r--r-- 1 user user   96 Sep 13 12:44 conda_build_config.yaml
-rw-r--r-- 1 user user 1584 Sep 13 12:45 LICENSE
-rw-r--r-- 1 user user 1064 Sep 13 12:10 meta.yaml

Generate an empty feedstock repository¶

Initialize a feedstock¶

Initialize a feedstock using conda-smithy:

$ cd ~/src/nsls-ii-forge
$ conda-smithy init <package-name>

A new directory ~/src/nsls-ii-forge/<package-name>-feedstock is created.

Replace conda-forge.yml in the feedstock directory with conda-forge.yml from the nsls-ii-forge’s event-model-feedstock package you cloned before:

$ cd ~/src/nsls-ii-forge
$ cp event-model-feedstock/conda-forge.yml <package-name>-feedstock/

Define Azure variables¶

$ export AZURE_ORG_OR_USER=nsls2forge
$ export AZURE_PROJECT_NAME=nsls2forge

For convenience, the above lines may be added to the ~/.bashrc file so that the environment variables are always available.

Create a GitHub repository and push files¶

$ cd ~/src/nsls-ii-forge/<package-name>-feedstock
$ git add .
$ git commit -m "Use nsls2forge's conda-forge.yml"
$ conda smithy register-github --organization nsls-ii-forge ./
$ git push -u upstream master

Enable CI on Azure Pipelines¶

$ conda smithy register-ci --organization nsls-ii-forge --without-circle \
--without-appveyor --without-travis --without-drone --feedstock_directory ./

Verify that CI was enabled on Azure Pipelines. Check for the following line in the output:

* nsls-ii-forge/<package-name>-feedstock has been enabled on azure pipelines

Rerender and push the feedstock repository¶

Rerender the feedstock¶

Create a new branch rerender:

$ git checkout -b rerender

Rerender the feedstock:

$ conda smithy rerender --feedstock_directory .

Note

From time to time conda-smithy may inform a user that its version is out-of-date, e.g.:

RuntimeError: conda-smithy version in root env (3.4.6) is out-of-date (3.4.7). Exiting.

Update conda-smithy to use the latest features:

$ conda install conda-smithy -c conda-forge

Commit the changes (the following command should be copied from the terminal output produced by the previous command):

$ git commit -m "MNT: Re-rendered with conda-build 3.18.9, conda-smithy 3.4.6, and conda-forge-pinning 2019.09.08"

Push changes to upstream:

$ git push -u upstream rerender

Create a pull request on GitHub¶

Open the GitHub page https://github.com/nsls-ii-forge/<package-name>-feedstock and create a pull request. In the pull request comments include a brief note and the link to the original repository of the package (PyPI, conda-forge or GitHub).

Closely examine the build results to ensure that the packages were built for all target platforms and Python versions and all tests passed successfully. Correct issues if necessary. Each time a change is made to configuration files, the feedstock must be rerendered and changes must be committed and pushed. Merge the pull request once all issues are fixed and the build statuses are green.

Once the PR build is finished, the new package or its new version will be uploaded to the Anaconda Cloud.

Known or possible issues¶

Tests fail for Linux due to a missing OpenGL¶

The solution is to place the file yum_requirements.txt into the recipes directory ~/src/nsls-ii-forge/<package-name>-feedstock/recipes, then rerender, commit and push changes.

Creating Packages¶

Directory structure¶

Set up the development environment¶

Install “conda-smithy”¶

Install Keybase¶

Configure “conda-smithy”¶

Create a root directory for packages¶

Clone the “event-model-feedstock” repository from the nsls-ii-forge GitHub org¶

Generate and edit the meta.yaml recipe configuration file¶

Generate a recipe from a PyPI package¶

Manually create a recipe or use the existing recipe¶

SHA256 digest¶

Prepare recipe files¶

Collect additional files¶

Generate an empty feedstock repository¶

Initialize a feedstock¶

Define Azure variables¶

Create a GitHub repository and push files¶

Enable CI on Azure Pipelines¶

Rerender and push the feedstock repository¶

Rerender the feedstock¶

Create a pull request on GitHub¶

Known or possible issues¶

Tests fail for Linux due to a missing OpenGL¶

Generate and edit the `meta.yaml` recipe configuration file¶