12.1. MAST development workflow¶
Last modified RJ 2017-04-13
12.1.1. I am a developer¶
12.1.1.1. Create a new branch¶
Go to the repository at http://git@github.com/uw-cmg/MAST
Create a new branch from dev.
12.1.1.2. Create new anaconda test environment with dependencies¶
In the instructions below, the test environment is called mast_myname_test.:
conda create -n mast_myname_test numpy scipy matplotlib pandas nose sphinx
source activate mast_myname_test
(mast_myname_test)[user@cluster ~]$ which conda
~/anaconda/envs/mast_myname_test/bin/conda
(mast_myname_test)[user@cluster ~]$ which python
~/anaconda/envs/mast_myname_test/bin/python
(mast_myname_test)[user@cluster ~]$ which pip
~/anaconda/envs/mast_myname_test/bin/pip
Install MAST dependencies by pip-installing MAST:
(mast_myname_test)[user@cluster ~]$ pip install --cert=~/anaconda/ssl/cacert.pem MAST
The cert tag may be important for finding newer versions etc.
If unsure, use pip -v -v -v (v for verbose)
You may also have to manually install ase and add it to PYTHONPATH
Now, you would actually want to develop on a git clone version of MAST rather than on the package-installed MAST.
So, unlink the package-installed MAST:
cd ~/anaconda/envs/mast_myname_test/lib/python2.7/site-packages/
mv MAST MAST-do-not-find
Clone a fresh copy and link to your own branch of MAST:
cd
git clone ssh://git@github.com/uw-cmg/MAST --branch myname mast_myname
The instructions above would put your MAST repository clone, with the branch name myname, into the folder //home/<user>/mast_myname
Add your repository to the activation script:
cd ~/anaconda/envs/mast_myname_test/bin
vi activate
...using a line like this:
export PYTHONPATH=$HOME/mast_myname:$PYTHONPATH
Make sure the repository is getting your branch version:
source deactivate
source activate mast_myname_test
python
>>> import MAST
>>> help(MAST)
Look for FILE to be //home/<user>/mast_myname/MAST/__init__.py
Run unit tests to check the installation. Occasionally nosetests will use the wrong python, so if that happens, instead of using the command nosetests, use:
cd //home/<user>/mast_myname/MAST/test
python -m nose
or:
python -m nose -v --nocapture
12.1.1.3. Code on your branch¶
Code on your branch in //home/<user>/mast_myname
Create your own unit tests in //home/<user>/mast_myname/MAST/test
Commit and push changes to your branch.:
git commit -a --message="commit message"
git push ssh://git@github.com/uw-cmg/MAST myname:myname
Run unit tests by:
source ~/anaconda/bin/activate mast_myname_test
cd ~/mast_myname/MAST/test
python -m nose
(Modify test/workflow_test/WORKFLOW_CONFIG and test/workflow_test/examples in order to run full workflow tests.)
You can also run unit tests singly, running only those tests directly affected by your changes. See nosetests documentation at python.org.
After development, run all unit tests.
Investigate failed tests. Repair code and/or tests as necessary.
When all tests have passed, commit and push to own branch.
12.1.1.4. Reconcile your branch with the development branch¶
Pull from dev to your local clone:
git pull ssh://git@github.com/uw-cmg/MAST dev
Address any merge errors.
Run unit tests.
Investigate failed tests. Repair code and/or tests as necessary, consulting with other developers as necessary.
Commit and push to your branch on github.
When all tests have passed, commit and push to dev.:
git push ssh://git@github.com/uw-cmg/MAST myname:dev
12.1.2. I am the release manager¶
12.1.2.1. Create a dev-test environment¶
Repeat the steps above under Create new anaconda test environment with dependencies with a new environment and the dev branch.
For the following instructions, assume your new environment is mast_dev_test, and that you cloned --branch dev into the folder mast_dev.
If any unit tests fail at this stage, assign any problems to developers. When developers have finished fixing tests, pull from dev again.
Each developer starts over from the Reconcile your branch with the development branch step.
12.1.2.2. Check documentation and upload to pypi¶
Change the version number in //home/<user>/mast_dev/MAST/_version.py
Edit the sphinx-build path in //home/<user>/mast_dev/MAST/doc/Makefile to correspond to //home/<user>/anaconda/envs/mast_dev_test/bin/sphinx-build.
Build the documentation:
source ~/anaconda/bin/activate mast_dev_test
(mast_dev_test)[user@cluster ]$ cd //home/<user>/mast_dev/MAST/doc
(mast_dev_test)[user@cluster ]$ make html
Investigate any bad builds and edit the rst files in /MAST/doc/source as necessary.
Copy the MAST/doc/build/html folder to your desktop.
Preview the html.
Modify MAST/doc/source pages as appropriate.
Rebuild and preview until the documentation is complete.
Zip the entire contents of html folder into a zip file by selecting all the files and folders in the html folder and compressing them (this has to be done from the desktop for some reason, or else it will not be readable by pythonhosted.org)
Log in to the pypi page at https://pypi.python.org/pypi/MAST At the bottom of the page, upload the zip into the Upload documentation section The upload may spin for a while, but it should work.
12.1.2.3. Package to the pypi test repository and test the package¶
Create a ~/.pypirc file that looks something like:
[distutils]
index-servers=
pypi
pypitest
[pypitest]
repository = https://testpypi.python.org/pypi
username = <username>
password = <pwd>
[pypi]
repository = https://pypi.python.org/pypi
username = <username>
password = <pwd>
Go to //home/<user>/mast_dev
Change MAST/_version.py to a use a dummy number (e.g. if the current version is 1.2.1, use 1.2.0.1)
Upload to the test pypi:
cd //home/<user>/mast_dev
python setup.py register -r pypitest sdist upload -r pypitest
If you are in a conda environment, deactivate it:
source deactivate
Make a new conda environment:
conda create -n mast_testpypi_test numpy scipy matplotlib pandas nose sphinx
source activate mast_testpypi_test
If there is a line adding a specific version of MAST in your ~/anaconda/envs/<new environment>/activate script, remove it, LOG OUT, and source activate your new environment. (Otherwise you will not install MAST from the pypi test repository.)
To install MAST from the test server:
pip install --cert ~/anaconda/ssl/cacert.pem pymatgen
pip install --cert ~/anaconda/ssl/cacert.pem custodian
pip install -i https://testpypi.python.org/pypi --cert ~/anaconda/ssl/cacert.pem MAST
(It is necessary to install some dependencies separately if they are not on the test server.)
Do not unlink MAST as you did for the mast_dev_test environment. This time, you want the package-installed MAST version.
Run unit tests.
Eventually this should work:
(mast_testpypi_test)[user@cluster ~]$ cd ~/anaconda/envs/mast_testpypi_test/lib/python2.7/site-packages/MAST/test
(mast_testpypi_test)[user@cluster ~]$ python -m nose
But in practice not all files get copied over, so use the test files from dev (which should be up to date, since you created the testpypi package from dev) but use this packaged version’s python and MAST.:
cd //home/<user>/mast_dev/mast_20160211_dev/MAST/test
python -m nose
If any unit tests fail, go back to the developers.
Each developer starts over from the Reconcile your branch with the development branch step.
When all unit tests have passed, the release manager starts over from I am the release manager.
If all tests pass, package to real repository.
12.1.2.4. Package to the real pypi repository¶
Change ~/mast_dev/MAST/_version.py to the correct version.
Package to the real pypi repository:
cd //home/<user>/mast_dev
python setup.py register -r pypi sdist upload -r pypi
Make a new environment as above and source it:
source deactivate
conda create -n mast_pypi_test numpy scipy matplotlib pandas nose sphinx
source activate mast_pypi_test
pip install --cert ~/anaconda/ssl/cacert.pem MAST
Run unit tests out of ~/mast_pypi_test.
Go back to developers if any unit tests fail. Nothing should fail from here, but it might.
If all tests pass, then only archiving, cleanup, and notifications are left.
12.1.2.5. Archiving¶
Sign in to github.com
- Draft a new release
- Use the same version number as in the pypi real repository as a tag (conventionally, v#.#.#)
- Use MAST-v#.#.# as the title
- Write a small description
- Publish release
- Save the .tar.gz file for this release
Sign in to zenodo.org
- Create a new Software upload with the .tar.gz file
12.1.2.6. Cleanup¶
- Remove all created environments using
conda env remove -n <environment name> - Remove all cloned MAST directories (except any that you use for personal coding)
- Remove any unneeded github branches that may have been created during the testing phase (e.g. if two developers create a mutual branch in order to test some code conflict)
12.1.2.7. Notification¶
Send out an email to the MAST development team and any core users. Done!
