Python Project, 2023 Style

This is an example repository intended to demonstrate best good standard practices for Python project.

Much of this project was put together by reading documentation from the various tools that are being used. Although I do use many of these tool in my Python projects I don't have a ton of miles on any specific project using all the features used with the specific configurations here.

So, there may be some issues here and there. Feel free to correct them and submit a merge request or simply file an issue.

You can also feel free to submit issues or merge requests with suggestions or comments regarding other useful tools and techniques that would be valuable to add.

Features

• Project management via hatch
  • Packaging
  • Build and tooling environments
  • Dynamic versioning using GIT tags
• Dependency pinning
  • pip-tools
• Linting and formatting
  • black
    • Formatting
  • ruff
    • Linting
  • mypy
    • Static type checker
  • isort
• Testing and coverage
  • pytest
  • coverage
• Docker image w/ rootless build
  • kaniko
• Pre-commit GIT hooks
  • pre-commit
• Gitlab CI/CD

The software itself it not intended to be particularly useful.

Other References:

• Python Packaging Authority PyPA
• Python Package Index PyPI
• Pip
  • "… the package installer for Python"
• Hatch
  • "… modern, extensible Python project manager"
• gitlab-ci.yml
• CI Variables
• Gitlab PyPI

Dynamic Versioning

Hatch is configured to manage the package version metadata specified using the __version__ variable in pyref/__init__.py.

The release.sh script is provided to help increment the package major, minor, or patch versions and create and put an associated GIT tag for the new version.

For example, if the current version is 0.1.0 and you run ./release.sh patch then:

1. The package version will be incremented to 0.1.1 in pyref/__init__.py
2. The version file pyref/__init__.py will be committed to git with the commit message "bump version" on the current branch
3. All commits and tags from the current branch will be pushed
   • git push --follow-tags

hatch

This configuration is the default configuration provided when creating a new hatch project using hatch new with few modifications.

The hatch configuration found in pyproject.toml supports a couple of different environments. There is the default environment, which is used when running any command using hatch run when not explicitly specifying the environment.

For example, to run the test-cov script defined in the default envinronment, .i.e., the [tool.hatch.envs.default.scripts] section of the configuration:

hatch run test-cov

This configuration provides default environment containing scripts for testing and coverage and the lint environment containing scripts for formatting and linting.

To run all linters to perform checking only:

hatch run lint:all

Or to apply fixes:

hatch run lint:fmt

CI/CD

Test Jobs

All the test jobs run in a default python:3.10 container. The container does not have hatch installed by default, so hatch is installed in each job. All jobs make use of the global cache and variables directives that will cause the pip cache to persist between builds.

There is also a specific lint job to check style and formatting. It has its own stage defined specifically to be before the other test stages. This results in all downstream jobs being skipped if the lint job fails.

Docker Images

This project creates a Docker Image built in CI without root permissions using kaniko. Building using kaniko is nice because it does not require root privileges, but I believe building this way precludes layer caching making the image builds much more heavy weight.

And alternative would be to build using Docker-in-Docker, however, I was not able to get this to work using the SSEC Shared Runner.

Images are built and pushed for commits to the main branch tagged using the first 8 characters of the GIT commit sha and also for any commit with a tag matching X.X.X.

The image build job (build:image) relies on artifacts produced upstream by the build:dist job. This dependency is reflected by the needs: ["build:dist"] configuration, which causes these 2 jobs to run serially event though they have the same stage.

Python Packages

Both the source and wheel distributions are uploaded to the project PyPI repository for all tagged X.X.X commits.

See the Gitlab PyPI docs for how to use the repository.

NOTE: This project is under my Gitlab user, not a Gitlab group. When a project is in a group any packages published to project repositores are also made available by the group level repository.