poetry_fix.py - Work around Poetry bugs¶
This script contains two fixes for Poetry bugs: one bug which manifests when the --no-dev
flag is passed to poetry install/update
and another which occurs when the --no-dev
flag isn’t passed. It doesn’t provide a fix to a third bug, discussed below
Invalid package METADATA¶
Per this issue, Poetry generates invalid package metadata for local path dependencies. For example, the last few lines of .venv/lib/python3.8/site-packages/runestone_poetry_project-0.1.0.dist-info/METADATA
contain:
Requires-Dist: pytz (>=2016.6.1)
Requires-Dist: requests (>=2.10.0)
Requires-Dist: rsmanage @ rsmanage
Requires-Dist: runestone
Requires-Dist: runestone-docker-tools @ docker
Requires-Dist: six (>=1.10.0)
Requires-Dist: sphinxcontrib-paverutils (>=1.17)
Requires-Dist: stripe (>=2.0.0,<3.0.0)
This causes an exception when running a command such as pip show click
:
ERROR: Exception:
Traceback (most recent call last):
File "/srv/web2py/applications/runestone/.venv/lib/python3.8/site-packages/pip/_vendor/pkg_resources/__init__.py", line 3021, in _dep_map
return self.__dep_map
File "/srv/web2py/applications/runestone/.venv/lib/python3.8/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2815, in __getattr__
raise AttributeError(attr)
AttributeError: _DistInfoDistribution__dep_map
… along with a long traceback of other chained exceptions.
Fixing the METADATA
file to be:
Requires-Dist: pytz (>=2016.6.1)
Requires-Dist: requests (>=2.10.0)
Requires-Dist: rsmanage @ file://rsmanage
Requires-Dist: runestone
Requires-Dist: runestone-docker-tools @ file://docker
Requires-Dist: six (>=1.10.0)
Requires-Dist: sphinxcontrib-paverutils (>=1.17)
Requires-Dist: stripe (>=2.0.0,<3.0.0)
… along with a similar fix to the METADATA
for bookserver_dev
allow pip
to run successfully.
TODO¶
Make this a poetry plugin, so it would auto-update this on any changes to the project’s
pyproject.toml
. It looks like plugins aren’t supported until v1.2.0, though.
Imports¶
These are listed in the order prescribed by `PEP 8`_.
Standard library¶
Third-party imports¶
Local application imports¶
None.
Fix for dev-dependencies
in subprojects¶
Given a main Poetry pyproject.toml
, these functions look for all subprojects included via path dependencies, creating additional subprojects named projectname-dev
in which the subproject’s dev-dependencies become dependencies in the newly-created subproject. This is a workaround for Poetry’s inability to install the dev dependencies for a sub project included via a path requirement. To use this, in the main project, do something like:
1[tool.poetry.dev-dependencies]
2sub = { path = "../sub", develop = true }
3sub-dev = { path = "../sub-dev", develop = true }
Create a project clone where the original project’s dev-dependencies are dependencies in the clone.
The path to the project.
Create a dev-only flavor.
If there are no dev-dependencies, there’s nothing to do. Otherwise, move them to dependencies.
Update the project name.
We don’t have a readme – if it exists, Poetry will complain about the missing file it references. Remove it if it exists.
Put the output in a project_name-dev/
directory.
Create a minimal project to make Poetry happy.
A dict of Poetry-specific values.
True to look at dependencies; False to look at dev-dependencies.
See project_path
.
See walked_paths_set
.
Given a pyproject.toml
, optionally create a dev dependencies project and walk all requirements with path dependencies.
The path where a pyproject.toml
exists.
walked_paths_set: a set of Paths already walked.
True if this is the root pyproject.toml
file – no dev dependencies will be created for it.
Avoid cycles and unnecessary work.
Process dependencies, if this is a Poetry project.
(Usually) process this file.
Core function: run the whole process on the pyproject.toml
in the current directory.
Fix for the main pyproject.toml
¶
This function updates the pyproject.toml
in the current directory by switching between a section named [tool.poetry.dev-dependencies]
when in development mode or [tool.no-poetry.dev-dependencies]
when not in development mode. This is because Poetry does not support either/or dependencies: either resolve dependency x in dev mode, or dependency y when not in dev mode. Instead, it takes a both/and approach: during its dependency resolution phase, it resolves ALL dependencies, then installs a subset (such all non-dev dependencies, or dev and non-dev dependencies). Quoting from the manual:
All dependencies must be compatible with each other across groups since they will be resolved regardless of whether they are required for installation or not (see Installing group dependencies).
Think of dependency groups as labels associated with your dependencies: they don’t have any bearings on whether their dependencies will be resolved and installed by default, they are simply a way to organize the dependencies logically.
Therefore, path based dev-dependencies break ‘install –no-dev’ when the directory does not exist. In addition, if a dependency exists both in the [tool.poetry.dependencies]
and the same dependency with a path in [tool.poetry.dev-dependencies]
sections, this version of Poetry will place the path in the resulting poetry.lock
file even when the --no-dev
option is passed, causing Poetry to install the dev version or fail if it’s not available.
As a workaround, this function renames the [tool.poetry.dependencies]
section, effectively hiding it, for --no-dev
option, and un-hides it otherwise. It then deletes poetry.lock
if it makes a change, ensuring that poetry will the run poetry update
with these changed dependencies.
Determine the current mode by setting has_dev
.
pyproject = Path("pyproject.toml")
pp_text = pyproject.read_text()
dev_section = "\n[tool.poetry.dev-dependencies]\n"
no_dev_section = "\n[tool.no-poetry.dev-dependencies]\n"
if dev_section in pp_text:
has_dev = True
elif no_dev_section in pp_text:
has_dev = False
else:
print(
f"Error: there is no [tool.(no-)poetry.dev-dependencies] section in {pyproject.resolve()}."
)
Update accordingly.
No update needed. We’re done.
Ideally, we’d run poetry update
here. However, we’re blocked from doing so by circular dependencies:
In a clean install, the command
poetry config virtualenvs.in-project true
has not executed yet.Running this command will first check the dependencies in the existing
poetry.lock
file and report that directories such as../BookServer
don’t exist. (Why does Poetry do this?)To update
poetry.lock
, we can runpoetry update
.But
poetry update
will update the wrong venv, sincepoetry config virtualenvs.in-project true
hasn’t run yet.Go to step 1.
So, just delete the lock file and let Poetry rebuild it; don’t complain if the file’s already been deleted.
CLI interface¶
This script works around Poetry bugs related to path dependencies.