Managing development environment, creating tests and monitoring test coverage, generating documentations, versioning the package. These are some common themes that I always find myself googling for solutions. In part 1 of the series, I am going to go through my current setup for environment management.
It is important to manage your development environment right. Without proper attention to this, you can end up with packages that are not re-distributable and only runnable on your machine. It can be also hard to onboard collaborators.
Having a separate environment for your package, instead of a global one, will prevent other projects from contaminating your project. For example, you might be developing a package that requires a specific version of
pandas, say 0.24, while another package has already onboarded with
pandas 1.0. Using global environments will cause conflicts and compatibility issues in one of the 2 packages.
To have an isolated environment, the first step is to use what Python called virtual environments.
My preferred package manager is pipenv. You can install it via
brew install pipenv
if you are on a Mac, or
pip install pipenv
otherwise. When you use
pipenv, it will generate 2 files. Pipfile and Pipfile.lock. You should focus on getting contents in Pipfile right. You can read more about them here. It looks like this
[[source]] name = "pypi" url = "https://pypi.org/simple" verify_ssl = true [dev-packages] versioneer = ">=0.18" twine = "*" [packages] requests = ">=2.22.0" pandas = "*" [requires] python_version = "3.7"
[dev-packages] are packages only required by package developers. Here, we have versioneer which is used for versioning the package, and twine for uploading the package to PYPI. [packages] are dependency needed for the package to function correctly.
To install package dependencies, run:
pipenv install [name-of-the-package]
To install development dependencies, run:
pipenv install [name-of-the-package] --dev
To activate the environment, run
and all your normal “python” commands now runs in the virtual environment with controlled dependencies. Some text editors and IDEs also supports pipenv natively. e.g. You can select pipenv as your interpreter in VS Code, and have the code runner run code through pipenv directly.
Also, make sure to install the package you are working on as a dev dependency, so your testing and other tooling can work in the virtual environment. Run
pipenv install -e . --dev
You should version control (using git or svn, I won’t judge you) both Pipfile and Pipfile.lock. When you moved to a new machine, to restore your current environment, in the project’s directory, run
and this will install all the specified files in Pipfile.
Lastly, if you want to generate
requirements.txt, you can run
pipenv run pip freeze > requirements.txt
Since Python 3.3, Python ships with its own venv package manager. We are not going to introduce venv here, but I thought the following meme is rather informative:
Dockerize Your Environment
Virtual Environments are great. However, there can still be dependencies on your OS that cannot be controlled through virtual environments. e.g. some packages cannot be installed correctly in Windows due to lack of built tools. If you are familiar with basic commands in Linux (e.g.
rm), I recommend you to dockerize your environment, and ship your project with a
Docker provides containerized solutions for environment management, and you can see containers as light weight virtual machines.
Dockerfile configures how the virtual machine should be setup.
Below is an example, where the docker container is configured to create a base environment based on continuum‘s miniconda, and then extended to install packages stored in the
requirement.txt file in the working directory. You’ll then have to build and run the container.
FROM continuumio/miniconda3 COPY requirements.txt /tmp/ RUN pip install --requirement /tmp/requirements.txt CMD [ "/bin/bash" ]
So why is this useful? As mentioned earlier, you’ll not only replicate package dependencies, but system dependencies via docker. This will basically create a Linux container for you develop your package, whether you are on MacOS, Windows, or other OS.
How can I make use of this? A docker container is like any other remote machine, and you’ll be able to attach it to your IDE so you can run your entire development within the container. For example, in VS Code, you can accomplish this via the Remote - Containers extension.
Here, I’ve created a minimal example. This assumes you are on a Mac (Linux should work too).
In the root directory, I’ve supplied a dockerfile, which pull’s from Python’s official docker image, and install pytest as a dev package via pipenv.
To run the docker container, you’ll need to build it once first. Simply run the
scripts/docker_build.sh file to do this.
Once the image is build, run
scripts/docker_run.sh, and this will start a docker container you can use for your development in the background.