Utilities¶
Utilities for handling feedstocks at nsls-ii-forge.
Here is a list of current functioning utilties:
check-results
Check the various parameters of a generated conda package
all-feedstocks
Clone/List all available feedstock repositories and get info from them
dashboard
Create a dashboard of package feedstocks for nsls-ii-forge
meta-utils
Extract and operate on information from meta.yaml feedstock files
graph-utils
Create a dependency graph of feedstock packages or query information from an existing one
auto-tick
Automatically update versions of packages by submitting pull requests for feedstock repositories
Note
A dependency graph is a directed acyclic graph that contains a node for each software package. The parents of a node (higher up in the graph) are packages that the node’s package requires to build, host, or run. See example usage below.
Paper & Poster¶
This project was supported as part of the Science Undergraduate Laboratory Internships (SULI) program at Brookhaven National Laboratory.
Below are links to the paper and poster outlining this project:
GitHub Authentication¶
Authentication is done through ~/.netrc and ~/.conda-smithy/github.token.
GitHub Access Token¶
First, create a personal access token for GitHub.
.netrc Setup¶
This may already be setup if you automatically authenticate on GitHub using git.
If not, create a file and open it for editing:
$ vi ~/.netrc
Place the following in that file with your GitHub username and access token.
machine github.com
login [USERNAME]
password [ACCESS_TOKEN]
To save the file and quit hit ESC, type wq and hit ENTER.
To make this file owner-readable only, use:
$ chmod 600 ~/.netrc
conda-smithy Setup¶
Cloning repositories requires authentication through conda-smithy. To install conda-smithy, run:
$ conda install -c conda-forge conda-smithy
Now place the token in the correct file:
$ vi ~/.conda-smithy/github.token
To save the file and quit hit ESC, type wq and hit ENTER.
To make this file owner-readable only, use:
$ chmod -Rv go-rwx ~/.conda-smithy
Note
Authentication using ~/.conda-smithy/github.token is only required
when cloning feedstock repositories.
conda-forge-tick Setup¶
Some utilities require functionality from the conda-forge-tick package.
The repository can be found here.
To install this package, use the following commands:
$ git clone --depth 1 https://github.com/regro/cf-scripts.git
$ cd ./cf-scripts
$ python setup.py install
Note
Cloning this repository is the only method of installing it since there is no tagged version available at the moment.
auto-tick Setup¶
For the auto-tick bot to submit pull requests on GitHub, it requires both a username and a token.
To do this, you need to set certain environment variables.
To use the nsls2forge GitHub account, export the following environment variables:
$ export GITHUB_USERNAME=nsls2forge
$ export GITHUB_TOKEN=[TOKEN]
Note
The token will not be publicly available, you can instead use your own
token and username and specify auto-tick run --fork. This will
fork the feedstock repository to the specified user and submit the
pull request from there.
Usage Examples¶
check-results¶
First, set up a conda environment.
$ conda create -n myenv
$ conda activate myenv
$ conda install -c defaults pip
Install nsls2forge packages from PyPI.
$ pip install event-model
You can check against a specific version of packages you are interested in with:
$ check-results -t version -p event-model -e 1.0
This will provide output like this:
The found version (1.15.2) of "event_model" is more or equal the expected version (1.0)
You can also check for packages from a specific channel with:
$ check-results -t channels -f conda-forge
This will list all installed packages from the conda-forge channel.
Note
The previous command will fail if packages are found from the specified channel.
To ignore this behavior use -i.
For more information on possible usage:
$ check-results -h
all-feedstocks¶
To list all feedstock repositories belonging to a GitHub organization use:
$ all-feedstocks -o nsls-ii-forge list
This will output a sorted list of repo names ending with -feedstock.
You can write these names to a file with the -w or --write flag.
You can include all archived feedstocks with the -a or --archived flag.
The default behavior is to ignore archived feedstocks.
Note
You can manually specify the username and personal access token for listing all feedstocks
with --username and --token instead of ~/.netrc.
To clone all feedstocks belonging to a GitHub organization, use:
$ all-feedstocks -o nsls-ii-forge clone
This will clone all feedstocks to the feedstocks/ directory in your current working directory.
To get information about Git and version information from currently cloned feedstock repos, use:
$ all-feedstocks info
For more information on possible usage:
$ all-feedstocks -h
$ all-feedstocks list -h
$ all-feedstocks clone -h
$ all-feedstocks info -h
dashboard¶
To create a dashboard of package feedstocks for nsls-ii-forge, use:
$ dashboard
Note
This uses ~/.netrc for GitHub authentication.
You can optionally provide a list of feedstock package names from a text file:
$ dashboard -n names.txt
You can also specify the output file name (default is README.md):
$ dashboard -w output.md
For more information on possible usage:
$ dashbord -h
meta-utils¶
To get any attribute of a feedstock packages meta.yaml file, use -g or --get:
Examples:
Getting the source url for event-model
$ meta-utils -o nsls-ii-forge -p event-model -g source url
source url: https://pypi.io/packages/source/e/event-model/event-model-1.15.2.tar.gz
Getting the package info for event-model
$ meta-utils -o nsls-ii-forge -p event-model -g package
package: {'name': 'event-model', 'version': '1.15.2'}
Getting the requirements to run event-model:
$ meta-utils -o nsls-ii-forge -p event-model -g requirements run
requirements run: ['python >=3.6', 'jsonschema', 'numpy']
To download a package from its source url, use:
$ meta-utils -o nsls-ii-forge -p event-model -d
Successfully downloaded https://pypi.io/packages/source/e/event-model/event-model-1.15.2.tar.gz
sha256: 31b6103801abcc3ebe099757bca7c9da9b4d535330acf0ba10d81c0753eb0e51
For more information on possible usage:
$ meta-utils -h
graph-utils¶
To create/update a dependency graph with new package feedstocks, use:
graph-utils make -o nsls-ii-forge
This will store the result in a JSON file graph.json.
To load the graph in your python script, use:
import json
import networkx as nx
j = json.load('graph.json')
graph = nx.node_link_graph(j)
To query information from the graph, use:
$ graph-utils info -p event-model -q depends_on
The following packages require event-model to be installed:
analysis
bluesky
bluesky-darkframes
databroker
databroker-pack
shed
suitcase-csv
suitcase-json-metadata
suitcase-jsonl
suitcase-mongo
suitcase-msgpack
suitcase-specfile
suitcase-tiff
xicam
xicam.xpcs
Total: 15
$ graph-utils info -p event-model -q depends_of
event-model requires the following packages to be installed:
jsonschema
numpy
pip
pytest
python
Total: 5
depends_on and depends_of are currently the only two types of queries.
To update nodes in the graph with new versions from their sources, use:
$ graph-utils update
This will pull new version numbers from various sources (PyPI, Github, etc.) and update each node with a tag “new_version”.
In the near future, there will be a bot that will handle packages with “new_version” greater than “version” and submit pull requests automatically to update the appropriate feedstock repostiories.
For more information on possible usage:
$ graph-utils -h
$ graph-utils make -h
$ graph-utils info -h
$ graph-utils update -h
auto-tick¶
To submit pull requests for feedstock repository version updates automatically, you will first have had to run at the very least:
$ graph-utils make
$ graph-utils update
This will provide the dependency graph with updated version numbers that the bot will use to decide
the correct ordering for updating packages. Please read the graph-utils section above
for more information on usage.
The bot can be run with:
$ auto-tick run --dry-run
$ auto-tick run
Note
You should always perform a dry run of the bot before running in full. This will display which package versions will be updated prior to making the change and submitting pull requests. If any errors occur during either the dry or full run, see the next section on cleaning.
To clean the current directory of all files related to the bot’s operation, use:
$ auto-tick clean
You will have to restart the process for making and updating the graph.
To output the status of all active migrations/pull requests, use:
$ auto-tick status
For more information on possible usage:
$ auto-tick -h
$ auto-tick run -h
$ auto-tick clean -h
$ auto-tick status -h