Python pip install Contextual Version Conflict

During the pip install of a package, we might end up at a message which reads as below

ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.

We recommend you use — use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.

During March 2020, Python package manager pip announced that pip dependency checks will be improved to remove inconsistencies in the package versions. And wow, that's coming to effect from Oct 2020.

In node.js and ruby, we have excellent package dependencies managers such that packages are being resolved individually in the most optimistic way. For example, node.js uses semver based logic to improve its efficiency in package dependency resolver.

Now we have happy news from pip, they are slowly adding the support for removing the inconsistency in the dependent packages conflict.

Benefits

1. No need for redundant packages with different versions in the same application
2. Improves the efficiency in the overall application

Steps to resolve the conflicts

You might have your list of dependencies in the setup.py or requirements.txt file, lets say we have the following structure in the requirements.txt

scipy==1.5.3
numpy==1.0.0
packageA==1.0.3
packageB==14.1.7

Here SciPy of version 1.5.3 depends on the latest version of NumPy(1.19.2) but the user has directly requested the NumPy version 1.0.0 directly. After Oct 2020, pip will throw an error saying conflict in package dependencies. We can resolve them as below,

• Loosen the package versions, we can set the NumPy version requirement as NumPy>1.0.0 This will help the pip to adjust the package dependent versions automatically.

scipy==1.5.3
numpy>=1.0.0
packageA==1.0.3
packageB==14.1.7

• Don't specify the package version at all, like below one

scipy==1.5.3
numpy
packageA==1.0.3
packageB==14.1.7

Here pip will automatically take care of the compatible version of NumPy

TIP: Never have unnecessary packages listed in your requirements.txt or setup.py file. They tend to cause a lot of conflicts.

• Sometimes we might have different levels of dependent packages, at this scenario try to mention the compatible package versions manually for the top one or two levels and leave the remaining levels unmentioned.
• Final case, if we are not able to find the compatible package versions for the application, reach out to the package maintainers in GitHub/lab and ask them to provide a compatible version of the package. Else, we can manually break down the package and use it as a small piece such that they are compatible with the application. (Kind of dependency hell)

If you want to check your application's compatibility against this feature,

1. Update the pip (line-1 in below script)
2. Use the options for 2020 resolver to check the package interdependencies and their conflicts for your application (line-2)

pip install --upgrade pip
pip install -r requirements.txt --use-feature=2020-resolver

Hope it helps :)

Reference: User Guide-Python