- Small Batches
- Posts
- Where do things go?
Where do things go?
Python project file/folder structure
[rating: beginner]
Historically, apart from a few specific files such as setup.py
, Python file & folder structure has been fairly loose & undefined. Posts such as this one on StackOverflow are very out of date - so don’t follow the guidance there.
More recently, as a result of PEP development & tool maturity the file system hierarchy has been stabilising a little. It’s a good time to
What’s the point?
Like PEP-8 style formatting, having a familiar & predictable file & folder structure across your projects will help your team know where to find the things they need to work on a project. They will be able to acquire the necessary context more reliably & quickly, which for the business means higher quality software & faster delivery.
Team members are less likely to accidentally duplicate work, & can be confident about where new work should go.
How does it work?
Here’s a minimal file & folder structure of a Python project called “my_project”.
my_project_repo/
├── src/
│ ├── my_project/
│ └── tests/
├── LICENSE
├── pyproject.toml
└── README.md
Fortunately you don’t need to know how to generate this hierarchy every time you need a new project. Your packaging tool will usually generate a basic hierarchy with some kind of init
subcommand, with some constraints.
If you’re using
setuptools
package manager then you are on your own. Use a separate tool likecookiecutter
to create a template & render the template each time you need it. Preferably don’t use setup tools at all.The folder structure above reflects that generated by the newer
pdm
&uv
package managers. Either of these is recommended.uv
is exceptionally fast, being developed in Rust.pdm
is slower, but still has excellent PEP compliance.poetry
is slightly older, less PEP compliant, & does it’s own thing a bit. For these reasons I don’t recommend using this package manager any more.
The core file that identifies this project as a Python project is now the pyproject.toml
file. Note that the repository name my_project_repo
does not necessarily have to be the same as the package name my_project
, but by convention this is usually the case - I have shown them as different simply for illustration purposes.
Formerly, using the setuptools
package manager in Python you might be aware of the setup.py
file. This practice is now deprecated because provisioning project configuration in an executable .py
file is poor security practice. There have been several historical examples of malicious & compromised open source projects doing surreptitious things in the setup.py file.
DON’T DO IT! (setup.py)
The tests
directory is in the src
directory to keep the project root directory smaller & easier to skim for information.
Here’s a more complete file/folder structure that I normally use for a working project.
my_project_repo/
├── build_harness/
├── dist/
├── docker/
├── docs/
├── features/
│ └── steps/
├── scripts/
├── src/
│ ├── my_project/
│ │ └── py.typed
│ └── tests/
│ ├── manual/
│ └── ci/
│ ├── integration/
│ └── unit/
├── LICENSE
├── (pdm|poetry|uv)\\.lock
├── pyproject.toml
├── README.md
├── setup.cfg
└── tasks.py
build_harness/
directory for CI, build & lifecycle management scripting. In the past I’ve used Makefiles, but more recently I prefer using invoke
for native Python task behaviour similar to make. The tasks.py
file is the Invoke entry point file, & any additional modules or packages are kept in the build_harness directory.
docker/
is where any Dockerfiles are stored, either for CI images, or production images of the project if the project is being distributed as a container.
docs/
is where configuration and source for Sphinx documentation is kept.
dist/
is a working directory for generated files such as wheel or sdist packages, code analysis reports, generated documentation, and so-on. This keeps
features/
feature & step files for Gherkin BDD
scripts/
sometimes projects need additional scripts for context management. Perhaps some Powershell for bootstrapping a dev environment on Windows, or bash for Linux. Hopefully you won’t need anything here, & Invoke tasks can be used for everything. Although I do like to have a single “bootstrap” script available to get all the dependencies installed in a virtual environment & ready for use.
setup.cfg
some tools haven’t yet migrated to use pyproject.toml configuration so still require the old-style setup.cfg. It’s less common now, but still needed sometimes.
In the tests
directory I make a further distinction of different kinds of tests.
manual tests - those developer tests that require human intervention while they are being run, or perhaps must be run against a live system.
ci tests - tests that do not require human intervention so that they can be run unattended with automation. Within CI tests it can be helpful to make a further distinction between integration & unit tests.
Because of widespread supply chain security issues, I consider lock files now to be an essential part of any package manager. They are not perfect, nor do they solve all the supply chain security problems. But they are a step forward on which further improvements can be added.
So don’t use a package manager that doesn’t support lock files. Another reason not to use setuptools.
References
https://stackoverflow.com/questions/193161/what-is-the-best-project-structure-for-a-python-application - MOSTLY OBSOLETE - REFERENCE ONLY
Want to modernise your Python workflow?
How about a 90 min session going over code you’re working on right now. Actionable recommendations on how to improve & make the development process faster & more effective. Get in touch!
Just a reminder…You’re probably receiving this email because you subscribed to the Small Batches newsletter. Don’t want to receive it anymore? Don’t file it to spam, use the unsubscribe link below! 👇 | Forwarded this post? Liked what you read & want more? |
I'm Russell 🐿️. A long time engineer discovering new life as a first-time solopreneur.
I build software delivery systems using Docker & Python automation that increase speed & reduce errors - saving money & improving profitability.
Reply