Minimal devpi For Local Python Development

Kronk Have Many Package

I am currently working on two internal two Python packages: an end-user application called openrelay, and its library dependency, openarc. In addition to this internal dependency chain, both packages have numerous external dependencies, and I would like everything to be installed nicely when I type pip install openrelay.

After struggling mightily with assorted hacks, I went searching for prior work. The common theme in all discussions was devpi, so I gave it a shot.

The experience was surprisingly intuitive, and I thought I'd write about it for future reference.

Setting up `devpi`

Pull the devpi suite in using pip:

pip-3.6 install devpi-server devpi-client --user

Initialize and start devpi-server. This will start a python package index on http://localhost:3141.

devpi-server --init --start

Check in on the status of the server:

ardentprayer% devpi-server --status
2019-03-20 07:03:53,744 INFO  NOCTX Loading node info from /home/kchoudhu/.devpi/server/.nodeinfo
2019-03-20 07:03:53,745 INFO  NOCTX wrote nodeinfo to: /home/kchoudhu/.devpi/server/.nodeinfo
server is running with pid 109

Out of the box, devpi creates a non-editable index root/pypi which simply proxies requests to external PyPI index and caches the result for future use:

pip-3.6 install Flask --index http://localhost:3141/root/pypi --user
Collecting Flask
  Downloading http://localhost:3141/root/pypi/+f/a08/0b744b7e345cc/Flask-1.0.2-py2.py3-none-any.whl (91kB)
    100% |████████████████████████████████| 92kB 24.4MB/s

Neat.

Create an internal-only index

A caching package proxy for external PyPI servers is cool and all, but our goal is to transparently distribute internal projects that are not available on external PyPI servers. To achieve this, we create a secondary index that inherits from root/pypi, and upload our development work to that.

Some admin work first. Create a user who will be able to upload to our secondary index:

devpi user -c distuser password=123

devpi login distuser --password=123

And create a new dev index:

devpi index -c dev bases=root/pypi volatile=True

The volatile keyword tells devpi that users will be able to upload to this index.

Upload projects to the internal-only index

Back to our initial problem: openrelay and openarc, and their external dependencies.

Both packages have coherent setup.py files, and I drop into their development directories and upload them to the devpi-server (for brevity's sake, only the output for openrelay is listed).

# cd ~/src/openrelay
# devpi upload
using workdir /tmp/devpi3
copied repo /usr/home/kchoudhu/src/openrelay/.git to /tmp/devpi3/upload/openrelay/.git
pre-build: cleaning /usr/home/kchoudhu/src/openrelay/dist
-->  /tmp/devpi3/upload/openrelay$ /usr/local/bin/python setup.py sdist --formats gztar
built: /usr/home/kchoudhu/src/openrelay/dist/openrelay-0.0.1.tar.gz [SDIST.TGZ] 15.535kb
register openrelay-0.0.1 to http://localhost:3141/distuser/dev/
file_upload of openrelay-0.0.1.tar.gz to http://localhost:3141/distuser/dev/

With both packages now uploaded to devpi, openrelay can be installed by issuing:

pip-3.6 install --index http://localhost:3141/distuser/dev openrelay --user

External dependencies are fetched and cached from PyPI, and openarc and openrelay were pulled in from devpi-server using one command.

Mission accomplished.

Configure `pip` to use `devpi`

Telling pip to use the new index with every invocation is tiresome, remove the need to do so by editing ~/.pip/pip.conf:

[global]
index-url = http://localhost:3141/distuser/dev

[search]
index = http://localhost:3141/distuser/dev

Do not put trailing slashes on the URLs; pip gets very testy indeed if you do.

Kronk Happy

I made developing multiple internal Python packages slightly little easier for myself, and heartily recommend that you follow in my footsteps. It is rare for a tool to improve productivity after, like, 6 commands, but here we are: devpi rules.

You should use it.