Going over a Python project structure up to a point where you can create a Python package.

Getting Started

We'll go through each file and directory of this project.

├── .dockerignore
├── .github
│   └── workflows
│       ├── docker-latest.yml
│       └── release.yml
├── .gitignore
├── Dockerfile
├── README.md
├── main.py
├── pyproject.toml
├── requirements.txt
├── scripts
│   ├── gh_create_release.sh
│   └── version_validation.sh
├── setup.cfg
├── setup.py
├── src
│   └── appy
│       ├── __init__.py
│       ├── __main__.py
│       ├── assets
│       │   ├── __init__.py
│       │   └── meirg-logo.jpg
│       ├── core
│       │   ├── __init__.py
│       │   └── app.py
│       └── utils
│           ├── __init__.py
│           ├── img_ascii.py
│           └── message.py
├── tests
│   ├── test_api.py
│   ├── test_greet.py
│   └── test_img_to_ascii.py
└── version

9 directories, 27 files

main.py and src/appy/__main__.py

The main.py file enables running the application as a module from the source code. It's possible to run a module from another module with runpy, and luckily this package is shipped with Python, so no installation is required.

The file which declares MARKDOWN_HASHf8d325e136c1b417b2cb6a0426fdca3fMARKDOWNHASH as a module is [src/appy/__main_\.py](https://github.com/unfor19/python-project/blob/master/src/appy/__main__.py), and the path from main.py to appy is src.appy.

See examples below

# Executing as a script - works as long as there are no relative imports of parent packages
# main.py uses runpy on src.appy > invokes __main__.py
(python-project) $ python main.py
My Path: python-project/main.py
Created the file: /Users/meirgabay/python-project/meirg-ascii.txt  Insert your name: willy wonka

Hello Willy Wonka, here's the cat fact of the day:
Jaguars are the only big cats that don't roar.

# Executing as a module - works because of the __main__.py file
# invokes __main__.py
(python-project) $ python -m src.appy
My Path: python-project/src/appy/__main__.py  Created the file: /Users/meirgabay/python-project/meirg-ascii.txt
Insert your name: willy wonka
Hello Willy Wonka, here's the cat fact of the day:

A cat has more bones than a human being; humans have 206 and the cat has 230 bones.


Contains packages (directories with __init__.py) and modules (*.py). This is also where you store files that you want to include in the package, such as *.jpg, *.json, etc. More about it in the next section - MANIFEST.in

You might be wondering why the only thing you see under src/ is appy/, and I'm glad that you find it interesting. I truly do.

It's very tricky to include __main__.py when your application package is in the project's root dir project-dir/appy. Using this structure, project-dir/src/appy guarantees that the file __main__.py will be included in the package. This is considered as a Best Practice by PyPa.


This file is optional and is relevant only if you want to include non-Python files in your package. For example, in this project, we have the file src/appy/assets/meirg-logo.jpg; this file is included in the package because it matched the expressions that were declared in MANIFEST.in, the exact pattern is graft src/appy

The below snippet is a straightforward and generic implementation that should fit most Python packages. To apply it in your Python package, replace appy with your package's name.

graft src/appy
include version main.py requirements.txt
global-exclude *.py[cod] __pycache__ *.so Dockerfile.* .dockerignore

To read more about MANIFEST.in, see the official docs - Using MANIFEST.in


The source of truth of the package's version. This value is updated automatically by a CI/CD process; more on that in the next blog posts. Its initial value is0.0.1.

src/appy/__init__.py and src/appy/*/__init__.py

When creating a Python package with setuptools, you'll need to add __init__.py to all of the packages, so the function find_packages will be able to import them automatically. Read more about it here in the docs - Listing whole packages.

The main purpose of src/appy/__init__.py file is to define the __version__ attribute of your package. This step will be covered in the following sections.

setup.cfg and setup.py

The file setup.cfg contains the package's static metadata, which includes all the information and instructions that are required by pip to install your package. The purpose of setup.py is to generate metadata values dynamically or perform a special task during build time. In this project, these are the tasks that setup.py is doing during build time:

  • Getting the package version from version file - this file is updated automatically as part of the CI/CD process; more on that in the next blog posts
  • Update src/appy/__init__.py to the current package version
  • Set the attributes version and download_url dynamically, according to the version

The important thing to remember here - setup.py and setup.cfg complement each other, so you shouldn't maintain the same attribute in both files. The separation should be according to the attribute; if it's never going to change, then set it in setup.cfg; if the attribute has a dynamic value, then it should be set in setup.py.
To learn more about setup.cfg and setup.py, go over them; I've added a bunch of comments in these files.

Package Discovery

The best way to automatically discover packages in the project (root dir) is to add the __init__.py file to every directory that is considered as a package. By doing so, we can use the options.packages.find in setup.cfg.

The below snippet instructs setuptools to search in the src directory, for __init__.py files. If a directory contains the __init__.py file, then it's considered as a package.

### Keep the same structure, Should NOT be changed
### Remember - create __init__.py in each directory that is a package
# [options] <--- we're here
packages = find:package_dir =    =srcinclude_package_data = True

where = src
### ------------------------------------------------------------------```


A list of packages, including version constraints, that are required for running the package. This file is usually generated in a Python Virtual Environment by executing the following command

$ (python-project) pip freeze > requirements.txt

The difference between install_packages and requirements.txt

  • install_requires: An abstraction of the required packages, without constraining to a specific version, but only setting the minimum required version, for example, requests>=2.24.0
  • requirements.txt: Constraining the required packages to a specific version for perfect compatibility. This is good for a package deployed as an application, for example, a Django web application. An example of how to constrain to a specific version requests==2.24.0


In this file, we declare which build tools, such as setuptools and wheel, are required for building this project. Read more about it in What the heck is pyproject.toml and PEP 518


The contents of this file will appear both in GitHub (or any other git host) and PyPi.


It's a great practice to write tests during the development process. This way, you can check that you haven't broken anything. Each time I add a new feature, I add tests; even though it sounds like overkill, it's totally worth it.

I'm using the built-in package unittest to test my application in this project.


Scripts that are not specifically related to the source code. These scripts are usually written in Bash, Python, Go, or any other scripting language. In this project, we have two scripts.

CI/CD, git And Docker Related

We'll cover the use of each file in the next blog posts.


This blog post is part of the Python Project series, and is based on this GitHub repository - unfor19/python-project.
The GitHub repo includes an example of a Python project, and Wiki Pages that describe the necessary steps for developing, creating and distributing a Python package.

Originally published at github.com/unfor19/python-project on 5 November, 2020

Recommended Posts

No comment yet, add your voice below!

Add a Comment

Your email address will not be published. Required fields are marked *