python ml skeleton project

generic skeleton for machine learning project with python, hydra, pytest, sphinx, github actions, etc. with dummy functionalities! It is mostly oriented geospatial projects

Why this project?

The goal of this project is to present a standard architecture of python repository/package including a full CiCd pipeline to document/test/deploy your project with standard methods of 2022. It can be used as starting point for any project without reinventing the wheel.

The code has no interest!

The code of this project is totally dummy: it makes simple mathematics operations like addition and subtration! The next iteration will make the opetations more interesting by using multi-layers perceptron! It will try to add a complete example of Hydra configuration.

In a close future, it will serve as a demonstrator by the example of a standard ML pipeline for experimentation and production

Installation

Install requirements

As Gdal dependencies are presents it's preferable to install dependencis via conda before installing the package:

  git clone https://github.com/samysung/python_ml_project_skeleton
  cd python_ml_project_skeleton/packaging
  conda env create -f package_env.yml

From pip:

pip install pmps
or pip install pmps==vx.x # for a specific version

python setup.py install

pip install -e .

pip install git+https://github.com/samysung/python_ml_project_skeleton

Project Architecture

├── CHANGELOG.rst
├── .codecov.yml
├── deploy
│   └── dockerfile
├── docs
│   ├── add.rst
│   ├── build.sh
│   ├── changelog.rst
│   ├── conf.py
│   ├── deploy.sh
│   ├── index.rst
│   ├── Makefile
│   ├── readme_link.md
│   └── _static
│       └── img
├── .github
│   └── workflows
│       ├── publish.yml
│       ├── test_code.yml
│       ├── test_docs.yml
│       ├── test_packaging.yml
│       └── test_publish.yml
├── .gitignore
├── LICENSE
├── packaging
│   ├── doc_env.yml
│   ├── doc_requirements.txt
│   ├── package_env.yml
│   ├── requirements.txt
│   ├── test_env.yml
│   └── test_requirements.txt
├── pmps
│   ├── api
│   │   ├── add.py
│   │   ├── __init__.py
│   │   └── subtract.py
│   ├── core
│   │   ├── add.py
│   │   ├── __init__.py
│   │   └── subtract.py
│   └── __init__.py
├── .pre-commit-config.yaml
├── .pylintrc
├── README.md
├── readthedocs.yml
├── setup.cfg
├── setup.py
├── tests
│   ├── api
│   │   ├── __init__.py
│   │   ├── test_add.py
│   │   └── test_subtract.py
│   └── __init__.py
└── VERSION

Architecture component overview

CI/CD pipeline

The first and essential goal is to have a skeleton quickly editable for a lot of use case projects with a big emphasis on continuous integration and continuous deployment. Here is a schematic view of the Ci/Cd pipeline targeted for open source python project, largely inspired by others well known projects:

DIAGRAM

Github Workflows

test code worflow (.github/workflows/test_code.yml):

Used to run unit test (and functionnal if implemented) tests on pull request events or push on main branch. It publishes coverage results on codecov.io. Use the packaging/test_env.yml conda environment file, github cache action and codecov/codecov-action

test docs workflow (.github/workflows/test_docs.yml):

Used to test the build of sphinx documentation. Run on pull request events or push on main branch. Use the packaging/doc_env.yml conda environment file, and the github cache action.

publish workflow (.github/workflows/publish.yml):

Used to publish the package on pypi, when a new tagged version or release is published. Use the packaging/package_env.yml conda environment file, github cache action github download and upload artifacts, and gh-action-pypi-publish.

test publish workflow (.github/workflows/test_publish.yml):

Same worflow as above, but on a test branch and test.pypi forge, for testing deployment improvement recipes

test packaging workflow (.github/workflows/test_packaging.yml):

Worflow actioned by CRON event (see crontab-guru), every n hours. Used to test that the package has been published and the lasted version is working.

Github workflows based on github webhooks or githu Apps

Some workflow works are handled by third party applications, like the readthe docs publication or the online pre-commit static analysis.

Pre-commit

Pre-commit action is launched via a github app (pre-commit.ci) on every commit made on remote. it's configured via the file pre-commit-config.yaml

Read-the-docs publication

Readthedocs publish new documentation version via a webhook subscribed for push and commit event. You can configure the type of push trigering the process in the readthedocs.org configuration section. See read the docs documentation for more detail.