Python Project - Create A Package
Going over a Python project structure, up to a point where you are able to create a Python package.
We'll go through each file and directory of this project.
. ├── .dockerignore ├── .github │ └── workflows │ ├── docker-latest.yml │ └── release.yml ├── .gitignore ├── Dockerfile ├── MANIFEST.in ├── README.md ├── main.py ├── pyproject.toml ├── requirements.txt ├── scripts │ ├── gh_create_release.sh │ └── version_validation.sh ├── setup.cfg ├── setup.py ├── src │ └── appy │ ├── __init__.py │ ├── __main__.py │ ├── assets │ │ ├── __init__.py │ │ └── meirg-logo.jpg │ ├── core │ │ ├── __init__.py │ │ └── app.py │ └── utils │ ├── __init__.py │ ├── img_ascii.py │ └── message.py ├── tests │ ├── test_api.py │ ├── test_greet.py │ └── test_img_to_ascii.py └── version 9 directories, 27 files
The file main.py: enables running the application from the source code. This file runs appy as a module with runpy. It's possible to run appy as a module because it contains the src/appy/__main__.py file.
# Excuting as a script - works as long as there are no relative imports of parent packages (python-project) $ python main.py My Path: python-project/main.py Created the file: /Users/meirgabay/python-project/meirg-ascii.txt Insert your name: willy wonka Hello Willy Wonka, here's the cat fact of the day: Jaguars are the only big cats that don't roar. # Excuting as a module - works because of the __main__.py file (python-project) $ python -m src.appy My Path: python-project/src/appy/__main__.py Created the file: /Users/meirgabay/python-project/meirg-ascii.txt Insert your name: willy wonka Hello Willy Wonka, here's the cat fact of the day: A cat has more bones than a human being; humans have 206 and the cat has 230 bones.
Contains packages (directories with
__init__.py) and modules (
*.py). This is also where you store files that you want to include in the package, such as
*.json, etc. More about it in the next section -
You might be wondering why the only thing you see under
src/ is src/appy/... It's best practice to place your application's package under
src/ since it makes the build process easier. The main issue that it solves including the file
__main__.py in the package. It's very tricky to include
__main__.py when your application package is in the project's root dir.
This file is optional and is relevant only if you want to include non-Python files in your package. For example, in this project, we have the file src/appy/assets/meirg-logo.jpg, this file is included in the package because it matched the expressions that were declared in MANIFEST.in, the exact pattern is
The below snippet is a very simple and generic implementation that should fit most Python packages. To apply it in your Python package, replace
appy with your package's name.
graft src/appy include version main.py requirements.txt global-exclude *.py[cod] __pycache__ *.so Dockerfile.* .dockerignore
To read more about MANIFEST.in see the official docs - Using MANIFEST.in
The source of truth of the package's version. This value is updated automatically by a CI/CD process, more on that in the next blog posts. Its initial value is
When creating a Python package with setuptools, you'll need to add
__init__.py to all of the
packages, so the function find_packages will be able to automatically import them. Read more about it here in the docs - Listing whole packages.
The main purpose of src/appy/__init__.py file is to define the
__version__ attribute of your package. This step will be covered in the following sections.
The file setup.cfg contains the package's static metadata, which includes all the information and instructions that are required by
pip to install your package. The purpose of setup.py is to generate metadata values dynamically, or perform a special task during build time. In this project, these are the tasks that setup.py is doing during build time:
- Getting the package version from version - this file is updated automatically as part of the CI/CD process, more on that in the next blog posts
- Update src/appy/__init__.py with the current package version- Set the attributes
download_urldynamically, according to the version
The important thing to remember here -
setup.cfg complement each other, so you shouldn't maintain the same attribute in both files. The separation should be according to the attribute, if it's never going to change then set it in
setup.cfg, if the attribute has a dynamic value then it should be set in
To learn more about setup.cfg and setup.py, go over them, I've added a bunch of comments in these files.
The best way to automatically discover packages in the project (root dir) is to add the
__init__.py file to every directory that is considered as a package. By doing so, we can use the
options.packages.find in setup.cfg. The below snippet instructs setuptools to search in the src directory, for
__init__.py files. If a directory contains the
__init__.py file, then it's considered as a package.
### Keep the same structure, Should NOT be changed ### Remember - create __init__.py in each directory that is a package # [options] <--- we're here packages = find:package_dir = =srcinclude_package_data = True [options.packages.find] where = src ### ------------------------------------------------------------------```
A list of packages including version constraints, that are required for running the package. This file is usually generated in a Python Virtual Environment, by executing the following command
$ (python-project) pip freeze > requirements.txt
The difference between install_packages and requirements.txt
- install_requires: An abstraction of the required packages, without constraining to a specific version, but only setting the minimum required version, for example,
- requirements.txt: Constraining the required packages to a specific version for perfect compatibility. This is good for a package that is deployed as an application, for example, a Django web application. An example of how to constrain to a specific version
The contents of this file will appear both in GitHub (or any other git host) and PyPi.
It's a great practice to write tests during the development process. This way you can check that you haven't broken anything. Each time that I add a new feature, I add tests, even though it sounds overkill, it is totally worth it. In this project, I'm using the built-in package unittest.
Scripts that are not specifically related to the source code. These scripts are usually written in Bash, Python, Go or any other scripting language. In this project, we have two scripts
- scripts/version_validation.sh: Makes sure that the provided release version accommodates with PEP 440 - Version Identification and Dependency Specification. It's being used as part of the CI/CD process, more on that in the next blog posts.
- scripts/gh_create_release.sh: Create a new GitHub release via command-line (terminal).
CI/CD, git And Docker Related
We'll cover the use of each file in the next blog posts.
- Dockerfile: The recipe for creating a Docker image of the package.
- .github/: Contains GitHub Actions workflows.
- .gitignore: Specifies intentionally untracked files that Git should ignore. More about in the git-scm
- .dockerignore: Same as
.gitignorebut for Docker build context.
- Well-structured Python projects
- Frameworks for creating a Python CLI
- The Click Framework is amazing! I used it in githubsecrets and frigga
- Read more about the available Python CLI frameworks in this great blog post - Building Beautiful Command Line Interfaces with Python
This blog post is part of the Python Project series, and is based on this GitHub repository - unfor19/python-project.
The GitHub repo includes an example of a Python project, and Wiki Pages that describe the necessary steps for developing, creating and distributing a Python package.
Originally published at github.com/unfor19/python-project on 5 November, 2020