Going over a Python project structure up to a point where you can create a Python package.
We'll go through each file and directory of this project.
. ├── .dockerignore ├── .github │ └── workflows │ ├── docker-latest.yml │ └── release.yml ├── .gitignore ├── Dockerfile ├── MANIFEST.in ├── README.md ├── main.py ├── pyproject.toml ├── requirements.txt ├── scripts │ ├── gh_create_release.sh │ └── version_validation.sh ├── setup.cfg ├── setup.py ├── src │ └── appy │ ├── __init__.py │ ├── __main__.py │ ├── assets │ │ ├── __init__.py │ │ └── meirg-logo.jpg │ ├── core │ │ ├── __init__.py │ │ └── app.py │ └── utils │ ├── __init__.py │ ├── img_ascii.py │ └── message.py ├── tests │ ├── test_api.py │ ├── test_greet.py │ └── test_img_to_ascii.py └── version 9 directories, 27 files
The main.py file enables running the application as a module from the source code. It's possible to run a module from another module with runpy, and luckily this package is shipped with Python, so no installation is required.
The file which declares
MARKDOWN_HASHf8d325e136c1b417b2cb6a0426fdca3fMARKDOWNHASH as a module is [src/appy/__main_\.py](https://github.com/unfor19/python-project/blob/master/src/appy/__main__.py), and the path from
main.py to appy is
See examples below
# Executing as a script - works as long as there are no relative imports of parent packages # main.py uses runpy on src.appy > invokes __main__.py (python-project) $ python main.py My Path: python-project/main.py Created the file: /Users/meirgabay/python-project/meirg-ascii.txt Insert your name: willy wonka Hello Willy Wonka, here's the cat fact of the day: Jaguars are the only big cats that don't roar. # Executing as a module - works because of the __main__.py file # invokes __main__.py (python-project) $ python -m src.appy My Path: python-project/src/appy/__main__.py Created the file: /Users/meirgabay/python-project/meirg-ascii.txt Insert your name: willy wonka Hello Willy Wonka, here's the cat fact of the day: A cat has more bones than a human being; humans have 206 and the cat has 230 bones.
Contains packages (directories with
__init__.py) and modules (
*.py). This is also where you store files that you want to include in the package, such as
*.json, etc. More about it in the next section -
You might be wondering why the only thing you see under
src/ is appy/, and I'm glad that you find it interesting. I truly do.
It's very tricky to include
__main__.py when your application package is in the project's root dir
project-dir/appy. Using this structure,
project-dir/src/appy guarantees that the file
__main__.py will be included in the package. This is considered as a Best Practice by PyPa.
This file is optional and is relevant only if you want to include non-Python files in your package. For example, in this project, we have the file src/appy/assets/meirg-logo.jpg; this file is included in the package because it matched the expressions that were declared in MANIFEST.in, the exact pattern is
The below snippet is a straightforward and generic implementation that should fit most Python packages. To apply it in your Python package, replace
appy with your package's name.
graft src/appy include version main.py requirements.txt global-exclude *.py[cod] __pycache__ *.so Dockerfile.* .dockerignore
To read more about MANIFEST.in, see the official docs - Using MANIFEST.in
The source of truth of the package's version. This value is updated automatically by a CI/CD process; more on that in the next blog posts. Its initial value is
When creating a Python package with setuptools, you'll need to add
__init__.py to all of the
packages, so the function find_packages will be able to import them automatically. Read more about it here in the docs - Listing whole packages.
The main purpose of src/appy/__init__.py file is to define the
__version__ attribute of your package. This step will be covered in the following sections.
The file setup.cfg contains the package's static metadata, which includes all the information and instructions that are required by
pip to install your package. The purpose of setup.py is to generate metadata values dynamically or perform a special task during build time. In this project, these are the tasks that setup.py is doing during build time:
- Getting the package version from version file - this file is updated automatically as part of the CI/CD process; more on that in the next blog posts
- Update src/appy/__init__.py to the current package version
- Set the attributes
download_urldynamically, according to the version
The important thing to remember here -
setup.cfg complement each other, so you shouldn't maintain the same attribute in both files. The separation should be according to the attribute; if it's never going to change, then set it in
setup.cfg; if the attribute has a dynamic value, then it should be set in
To learn more about setup.cfg and setup.py, go over them; I've added a bunch of comments in these files.
The best way to automatically discover packages in the project (root dir) is to add the
__init__.py file to every directory that is considered as a package. By doing so, we can use the
options.packages.find in setup.cfg.
### Keep the same structure, Should NOT be changed ### Remember - create __init__.py in each directory that is a package # [options] <--- we're here packages = find:package_dir = =srcinclude_package_data = True [options.packages.find] where = src ### ------------------------------------------------------------------```
A list of packages, including version constraints, that are required for running the package. This file is usually generated in a Python Virtual Environment by executing the following command
$ (python-project) pip freeze > requirements.txt
The difference between install_packages and requirements.txt
- install_requires: An abstraction of the required packages, without constraining to a specific version, but only setting the minimum required version, for example,
- requirements.txt: Constraining the required packages to a specific version for perfect compatibility. This is good for a package deployed as an application, for example, a Django web application. An example of how to constrain to a specific version
The contents of this file will appear both in GitHub (or any other git host) and PyPi.
It's a great practice to write tests during the development process. This way, you can check that you haven't broken anything. Each time I add a new feature, I add tests; even though it sounds like overkill, it's totally worth it.
I'm using the built-in package unittest to test my application in this project.
Scripts that are not specifically related to the source code. These scripts are usually written in Bash, Python, Go, or any other scripting language. In this project, we have two scripts.
- scripts/version_validation.sh: Makes sure that the provided release version accommodates with PEP 440 - Version Identification and Dependency Specification. It's being used as part of the CI/CD process, more on that in the next blog posts.
- scripts/gh_create_release.sh: Create a new GitHub release via command-line (terminal).
CI/CD, git And Docker Related
We'll cover the use of each file in the next blog posts.
- Dockerfile: The recipe for creating a Docker image of the package.
- .github/: Contains GitHub Actions workflows.
- .gitignore: Specifies intentionally untracked files that Git should ignore. More about in the git-scm
- .dockerignore: Same as
.gitignorebut for Docker build context.
- Well-structured Python projects
- Frameworks for creating a Python CLI
- The Click Framework is amazing! I used it in githubsecrets and frigga
- Read more about the available Python CLI frameworks in this great blog post - Building Beautiful Command Line Interfaces with Python
This blog post is part of the Python Project series, and is based on this GitHub repository - unfor19/python-project.
The GitHub repo includes an example of a Python project, and Wiki Pages that describe the necessary steps for developing, creating and distributing a Python package.
Originally published at github.com/unfor19/python-project on 5 November, 2020