Python Project - Create A Package

Goal

Going over a Python project structure, up to a point where you are able to create a Python package.

Getting Started

We'll go through each file and directory of this project.

.
├── .dockerignore
├── .github
│   └── workflows
│       ├── docker-latest.yml
│       └── release.yml
├── .gitignore
├── Dockerfile
├── MANIFEST.in
├── README.md
├── main.py
├── pyproject.toml
├── requirements.txt
├── scripts
│   ├── gh_create_release.sh
│   └── version_validation.sh
├── setup.cfg
├── setup.py
├── src
│   └── appy
│       ├── __init__.py
│       ├── __main__.py
│       ├── assets
│       │   ├── __init__.py
│       │   └── meirg-logo.jpg
│       ├── core
│       │   ├── __init__.py
│       │   └── app.py
│       └── utils
│           ├── __init__.py
│           ├── img_ascii.py
│           └── message.py
├── tests
│   ├── test_api.py
│   ├── test_greet.py
│   └── test_img_to_ascii.py
└── version

9 directories, 27 files

main.py and src/appy/__main__.py

The file main.py: enables running the application from the source code. This file runs appy as a module with runpy. It's possible to run appy as a module because it contains the src/appy/__main__.py file.

# Excuting as a script - works as long as there are no relative imports of parent packages
(python-project) $ python main.py  My Path: python-project/main.py
Created the file: /Users/meirgabay/python-project/meirg-ascii.txt  Insert your name: willy wonka

Hello Willy Wonka, here's the cat fact of the day:
Jaguars are the only big cats that don't roar.

# Excuting as a module - works because of the __main__.py file
(python-project) $ python -m src.appy
My Path: python-project/src/appy/__main__.py  Created the file: /Users/meirgabay/python-project/meirg-ascii.txt
Insert your name: willy wonka
Hello Willy Wonka, here's the cat fact of the day:

A cat has more bones than a human being; humans have 206 and the cat has 230 bones.

src/

Contains packages (directories with __init__.py) and modules (*.py). This is also where you store files that you want to include in the package, such as *.jpg, *.json, etc. More about it in the next section - MANIFEST.in

You might be wondering why the only thing you see under src/ is src/appy/... It's best practice to place your application's package under src/ since it makes the build process easier. The main issue that it solves including the file __main__.py in the package. It's very tricky to include __main__.py when your application package is in the project's root dir.

MANIFEST.in

This file is optional and is relevant only if you want to include non-Python files in your package. For example, in this project, we have the file src/appy/assets/meirg-logo.jpg, this file is included in the package because it matched the expressions that were declared in MANIFEST.in, the exact pattern is graft src/appy

The below snippet is a very simple and generic implementation that should fit most Python packages. To apply it in your Python package, replace appy with your package's name.

graft src/appy
include version main.py requirements.txt
global-exclude *.py[cod] __pycache__ *.so Dockerfile.* .dockerignore

To read more about MANIFEST.in see the official docs - Using MANIFEST.in

version

The source of truth of the package's version. This value is updated automatically by a CI/CD process, more on that in the next blog posts. Its initial value is0.0.1.

src/appy/__init__.py and src/*/__init__.py

When creating a Python package with setuptools, you'll need to add __init__.py to all of the packages, so the function find_packages will be able to automatically import them. Read more about it here in the docs - Listing whole packages.

The main purpose of src/appy/__init__.py file is to define the __version__ attribute of your package. This step will be covered in the following sections.

setup.cfg and setup.py

The file setup.cfg contains the package's static metadata, which includes all the information and instructions that are required by pip to install your package. The purpose of setup.py is to generate metadata values dynamically, or perform a special task during build time. In this project, these are the tasks that setup.py is doing during build time:

  • Getting the package version from version - this file is updated automatically as part of the CI/CD process, more on that in the next blog posts
  • Update src/appy/__init__.py with the current package version- Set the attributes version and download_url dynamically, according to the version

The important thing to remember here - setup.py and setup.cfg complement each other, so you shouldn't maintain the same attribute in both files. The separation should be according to the attribute, if it's never going to change then set it in setup.cfg, if the attribute has a dynamic value then it should be set in setup.py.
To learn more about setup.cfg and setup.py, go over them, I've added a bunch of comments in these files.

Package Discovery

The best way to automatically discover packages in the project (root dir) is to add the __init__.py file to every directory that is considered as a package. By doing so, we can use the options.packages.find in setup.cfg. The below snippet instructs setuptools to search in the src directory, for __init__.py files. If a directory contains the __init__.py file, then it's considered as a package.

### Keep the same structure, Should NOT be changed
### Remember - create __init__.py in each directory that is a package
# [options] <--- we're here
packages = find:package_dir =    =srcinclude_package_data = True

[options.packages.find]
where = src
### ------------------------------------------------------------------```

requirements.txt

A list of packages including version constraints, that are required for running the package. This file is usually generated in a Python Virtual Environment, by executing the following command

$ (python-project) pip freeze > requirements.txt

The difference between install_packages and requirements.txt

  • install_requires: An abstraction of the required packages, without constraining to a specific version, but only setting the minimum required version, for example, requests>=2.24.0
  • requirements.txt: Constraining the required packages to a specific version for perfect compatibility. This is good for a package that is deployed as an application, for example, a Django web application. An example of how to constrain to a specific version requests==2.24.0

pyproject.toml

In this file we declare which build tools, such as setuptools and wheel, are required for building this project. Read more about it in What the heck is pyproject.toml and PEP 518

README.md

The contents of this file will appear both in GitHub (or any other git host) and PyPi.

tests/

It's a great practice to write tests during the development process. This way you can check that you haven't broken anything. Each time that I add a new feature, I add tests, even though it sounds overkill, it is totally worth it. In this project, I'm using the built-in package unittest.

scripts/

Scripts that are not specifically related to the source code. These scripts are usually written in Bash, Python, Go or any other scripting language. In this project, we have two scripts

CI/CD, git And Docker Related

We'll cover the use of each file in the next blog posts.

References


This blog post is part of the Python Project series, and is based on this GitHub repository - unfor19/python-project.
The GitHub repo includes an example of a Python project, and Wiki Pages that describe the necessary steps for developing, creating and distributing a Python package.


Originally published at github.com/unfor19/python-project on 5 November, 2020

Recommended Posts

No comment yet, add your voice below!


Add a Comment

Your email address will not be published. Required fields are marked *