Create Python dependencies repo health dashboard

[jmbowman]

For the Python 3.8 and Django 3.2 upgrades, we manually created a spreadsheet to track which dependencies were known to already work with the target versions: https://docs.google.com/spreadsheets/d/19-BzpcX3XvqlazHcLhn1ZifBMVNund15EwY3QQM390M/edit#gid=0 . This worked, but required a lot of manual effort to flesh it out and keep it accurate. For the Django 4.2 and Python 3.11 upgrades, we'd like to automate this better. Suggested approach:

1. Define repo health checks for the relevant bits of information about each dependency (if such checks don't already exist); suggestions listed below.
2. Support a cache of information about dependencies which can be used to override the calculated values which would otherwise be used on the dashboard. For example, a package may have officially supported Python 3.11 for the past 3 releases but forgotten to add the Trove classifier until after the latest release. Try to implement this generically enough that it can easily be used to override other checks in the future as well.
3. Define a relevant subset of all the repo health checks to use as an upgrade project dashboard.
4. Create a GitHub Actions Workflow which will fetch the desired information for each target repository daily. We may need to create a manual repository enumeration for this, although we should include some kind of validation that all Python packages listed as dependencies anywhere in the main repo health dashboard are included in that list.

Checks we will probably want include:

• Which version of the dependency added Python 3.11 support?
• Does the dependency use Django?
• Which version of the dependency added Django 4.0 support?
• Which version of the dependency added Django 4.1 support?
• Which version of the dependency added Django 4.2 support?
• Does the dependency have the correct trove classifiers for each of Python 3.11 / Django 4.0 / Django 4.1 / 4.2 support in its git repository? (If not, we may want to help the upstream maintainer fix that even if we've manually confirmed that it works.)
• Does the latest release of the dependency have each of those trove classifiers?
• If an Open edX issue has been created to get the dependency updated for Python 3.11 and/or Django 4.2 support, where is it? (This would probably have to be done as a lookup in a data file local to the check implementation, or maybe a public readable Google Sheet.)
• What is the URL of the best available changelog for the package? (Which we currently track manually and incompletely at https://openedx.atlassian.net/wiki/spaces/AC/pages/1001521320/Python+Package+Changelogs )
• Is the package available on PyPI? (A few are installed directly from GitHub instead.)

I may be forgetting other things we'll also need, feel free to suggest anything I've missed or other ideas for improvement.

[jmbowman]

Having the changelog URLs available in the data files would enable later work to make them available as links in the auto-generated "make upgrade" PR information.

[jmbowman]

I found this in my notes for the current repo health checks that we'd probably want to include for a Python dependencies dashboard:

• repo_name
• TIMESTAMP
• github.branch_count
• github.pulls_count
• org_name
• dependencies.pypi.*
• django_packages.*
• requires.django
• setup_py.*

[zubairshakoorarbisoft]

Here are the few updates on this issue:

• We will write s separate job to keep dependencies urls updated, for that we have write a script which is in repo-health-data, Pr can be review here
• We have refactored common checks code by making a reusable function and calling them in multiple checks, and moved some fixtures of repo_health to pytest-repo-health so that those can be accessible globally by multiple test sessions. PR for the new checks for dependencies can be review here
• Here is the sample yaml data file we're generating for dependencies
• Here is the dependencies dashboard csv file making using above yaml data files

Please let me know if you have some suggestions to improve it.

CC: @iamsobanjaved @jmbowman @awais786

[UsamaSadiq]

The dashboard workflow has been created. A follow up issue will be created if we need to work further on this issue hence closing this issue for now.