Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
127 commits
Select commit Hold shift + click to select a range
82e3dfa
Migrated to using upstream keystone oidc plugin
QuanMPhm Mar 13, 2024
e1a40f6
Merge pull request #150 from QuanMPhm/149/migrate_keycloak
knikolla Mar 21, 2024
f83faaa
Remove OIDC plugin from Openstack functional test
QuanMPhm Mar 21, 2024
ce57a50
Merge pull request #152 from QuanMPhm/149/migrate_keycloak
knikolla Mar 26, 2024
93cf8f2
Add institution specific code to invoice
knikolla Mar 27, 2024
f737f7b
Merge pull request #153 from knikolla/iss
knikolla Mar 27, 2024
9bcf328
Fix missing dependency, incorrect string quotes and argparse method
knikolla Apr 10, 2024
b4464cf
Merge pull request #155 from knikolla/fix_s3
QuanMPhm Apr 11, 2024
5b07de8
Added management command to migrate field of sciences
QuanMPhm Apr 10, 2024
f2db32e
Merge pull request #154 from QuanMPhm/migrate_fos
knikolla Apr 12, 2024
db048fd
Return PI email in storage invoices
knikolla May 1, 2024
0794055
Merge pull request #156 from knikolla/pi_email_in_storage_invoices
knikolla May 2, 2024
fe4f9a5
validate: catch and log exceptions from set_quota
jtriley May 13, 2024
e4a225b
openshift: handle zero values when setting quotas
jtriley May 13, 2024
9f8d2b7
openstack: set minimum for object quota to 1
jtriley May 13, 2024
8e7459f
test openstack object storage quota
jtriley May 14, 2024
af75e98
test validate_allocations fix for object quota = 0
jtriley May 14, 2024
92938e1
Merge pull request #158 from nerc-project/rgw-disable-zero-quota
jtriley May 14, 2024
8b3b651
validate: fix object store quota value comparison
jtriley May 14, 2024
ae311f8
Merge pull request #160 from nerc-project/fix-validate-obj-quota-comp…
jtriley May 14, 2024
4a4d347
renamed the attributes with GiB and MiB removing GB and MB
Milstein May 23, 2024
9e2c625
Revert the OpenStack swift attribute change
Milstein May 23, 2024
26b9890
OpenStack Volume Quota --> OpenStack Number of Volumes Quota
Milstein May 23, 2024
6221350
OpenStack Number of Volumes Quota
Milstein May 24, 2024
4852752
Merge pull request #164 from nerc-project/rename_attributes
knikolla May 28, 2024
775afd2
Added rounding to Cost and Rate for storage invoice
QuanMPhm May 15, 2024
6a2d53a
Merge pull request #161 from QuanMPhm/157/invoice_rounding
knikolla May 31, 2024
f42f028
Convert Swift quotas from GB to GiB
QuanMPhm Jun 12, 2024
f7a4dc2
Merge pull request #168 from QuanMPhm/162/covert_swift_quotas
QuanMPhm Jun 17, 2024
9885c12
Fix missing simplejson package
knikolla Aug 1, 2024
58d575e
Update microshift job to 22.04
knikolla Aug 1, 2024
a9a7b01
Merge pull request #173 from knikolla/fix_ci_2024_08
knikolla Aug 1, 2024
3bc63e5
Added test cases to ensure correct billing with allocation change req…
QuanMPhm Jul 31, 2024
e523675
Fixed bug for billing with change requests
QuanMPhm Aug 1, 2024
07afcf0
Merge pull request #171 from QuanMPhm/bug/CRs
knikolla Aug 2, 2024
1518986
Implemented 'nerc-rates' for storage invoicing
QuanMPhm Jun 13, 2024
591aa23
Merge pull request #169 from QuanMPhm/167/use_nerc_rates
knikolla Aug 26, 2024
0c05d81
Support excluding time intervals during storage invoicing
QuanMPhm Jul 30, 2024
ec453be
Fix typo
QuanMPhm Jul 30, 2024
ce7c8e1
Merge pull request #174 from QuanMPhm/166/exclude_intervals
knikolla Oct 2, 2024
0a745f2
Intergrated ESI, allowing users to allocate ESI Network quotas
QuanMPhm Mar 10, 2024
b304508
Fix step name in unit test workflow
larsks Nov 1, 2024
00d90e1
Have dependabot manage gitub action versions
larsks Nov 1, 2024
3d5160d
Merge pull request #179 from larsks/fix/workflow-label
knikolla Nov 1, 2024
fab0326
Avoid interactive prompts during unit tests
larsks Nov 1, 2024
09c6c6a
Upgrade minimum version of coldfront
larsks Nov 1, 2024
63a6c74
Merge pull request #180 from larsks/fix/dependabot-for-actions
larsks Nov 1, 2024
f48eba0
Bump actions/setup-python from 1 to 5
dependabot[bot] Nov 1, 2024
e11aced
Merge pull request #178 from larsks/fix/interactive-prompt-during-tests
knikolla Nov 1, 2024
f3d0574
Merge pull request #181 from nerc-project/dependabot/github_actions/a…
knikolla Nov 1, 2024
9ee2857
Bump actions/checkout from 2 to 4
dependabot[bot] Nov 1, 2024
4626b80
Merge pull request #182 from nerc-project/dependabot/github_actions/a…
larsks Nov 1, 2024
f5cdea3
Add missing shebang (#!) lines
larsks Nov 2, 2024
5b6a5cf
Merge pull request #183 from larsks/fix/missing-shebang
larsks Nov 2, 2024
c5002a5
Merge pull request #146 from QuanMPhm/135/ESI_Support
knikolla Dec 13, 2024
ba8aae8
Fix incorrect code block split into 2 lines
knikolla Jan 31, 2025
4e8e6c7
Merge pull request #189 from knikolla/fix/broken_code
knikolla Feb 5, 2025
103277a
Fix CI
knikolla Feb 5, 2025
c61fd6e
Merge pull request #192 from knikolla/fix/devstack_2025
knikolla Feb 7, 2025
6752900
Allow direct communication to Openshift API through `get_federated_user`
QuanMPhm Oct 29, 2024
6356e64
Support hours when providing outage intervals in billing
knikolla Jan 31, 2025
8789c47
Make openshift and openstack gb rates optional
knikolla Feb 10, 2025
02e036c
Add testing for CLI call with exclude
knikolla Feb 10, 2025
05d9739
Lazily load nerc-rates to avoid Python 3.9 incompat
knikolla Feb 10, 2025
3cc766c
Merge pull request #190 from knikolla/feature/ignore_hours
knikolla Feb 11, 2025
2ad835f
Merge pull request #177 from QuanMPhm/175/merge_get_federated
QuanMPhm Mar 25, 2025
e90227e
Moved inline CI script into appropriate ci script file
QuanMPhm Apr 4, 2025
4b24fae
Merge pull request #204 from QuanMPhm/198/move_inline_script
QuanMPhm Apr 8, 2025
d38528b
Upgrade workflows to use Python 3.12
QuanMPhm Mar 7, 2025
d275b23
Suppress `InsecureRequestWarning` warnings for Openstack functional t…
QuanMPhm Apr 16, 2025
fcebc93
Merge pull request #197 from QuanMPhm/update_python
knikolla Apr 16, 2025
49bca1d
Fixed integration with Openshift API
QuanMPhm Apr 5, 2025
87ad16a
Allow direct communication with Openshift Quota API
QuanMPhm Apr 4, 2025
5d0d1e9
Merge pull request #205 from QuanMPhm/187/acct_quota
knikolla May 6, 2025
dd7b21f
Updated version pin for account manager in workflow
QuanMPhm Feb 25, 2025
4e216a5
Implemented a new resource class for Openshift virtualization
QuanMPhm Feb 24, 2025
c533baa
Merge pull request #203 from QuanMPhm/add_vm_gpu_quotas
knikolla May 6, 2025
9242463
Refactored `validate_allocation` to use `tasks.UNIT_QUOTA_MULTIPLIERS`
QuanMPhm Apr 3, 2025
c9ef7a1
Merge pull request #201 from QuanMPhm/refactor/validate_allo
QuanMPhm May 7, 2025
4ea0599
Allow adding default labels to Openshift allocations
QuanMPhm Apr 20, 2025
c5ac22c
Merge pull request #211 from QuanMPhm/196/add_label
knikolla May 8, 2025
1d242e7
Remove python 3.9 checks from CI tests
KelvinLinBU May 8, 2025
34ef7d2
Remove memcached pin
knikolla May 12, 2025
a6d4c7d
Merge pull request #216 from knikolla/remove_memcached
knikolla May 12, 2025
ac89456
Merge pull request #215 from KelvinLinBU/updatepyversion
QuanMPhm May 20, 2025
e3c4a01
209: add pre-commit hooks
sudoCommit Apr 21, 2025
fd81008
Merge pull request #210 from sudoCommit/feature/#209
QuanMPhm May 28, 2025
f9a9ef1
Applied pre-commit hook
QuanMPhm May 28, 2025
d776019
Merge pull request #220 from QuanMPhm/209/linter
knikolla May 29, 2025
4063cca
Allow direct communication to Openshift Projects API
QuanMPhm May 20, 2025
c9f19f6
Explicitly request type when using `nerc_rates`
KelvinLinBU Apr 24, 2025
cc6b1fa
Merge pull request #212 from KelvinLinBU/update-get-value-at
knikolla May 30, 2025
1c2a724
Support complex resource names loading env vars
KelvinLinBU May 20, 2025
934b06a
Added `Cluster Name` column to storage invoice
QuanMPhm May 30, 2025
5e2efea
Merge pull request #218 from KelvinLinBU/robust-conversions
knikolla Jun 13, 2025
3aaaff3
Resolve nits regarding direct communications to Openshift API
KelvinLinBU Jun 19, 2025
fcd7dd1
Merge pull request #227 from KelvinLinBU/nits
knikolla Jun 21, 2025
cee3d22
Allow direct communication to Openshift Users API
QuanMPhm May 30, 2025
31000f3
Merge pull request #226 from QuanMPhm/221/cluster_name
knikolla Jun 30, 2025
2f50bff
Fix Cluster Name in invoice
knikolla Jul 1, 2025
7d5fdd4
Merge pull request #228 from knikolla/fix/cluster_name_header
knikolla Jul 1, 2025
8bdb720
Allow direct communication to Openshift RBAC API
QuanMPhm Jul 23, 2025
df704e8
Allow direct communication to Openshift Users API
QuanMPhm May 30, 2025
8958d10
Merge pull request #219 from QuanMPhm/185/openshift_projects
QuanMPhm Aug 5, 2025
e950e83
Merge pull request #224 from QuanMPhm/184/openshift_users
QuanMPhm Aug 5, 2025
7e66279
Merge pull request #229 from QuanMPhm/186/openshift_rbac
QuanMPhm Aug 5, 2025
11587a3
Implemented feature to add new attributes to existing Openshift alloc…
QuanMPhm Apr 3, 2025
717d770
Bump actions/checkout from 4 to 5
dependabot[bot] Aug 12, 2025
3cacbc6
Fix calculate_storage_gb_hours
QuanMPhm Aug 12, 2025
05c9e2b
Merge pull request #235 from QuanMPhm/fix/calc_storage
QuanMPhm Aug 12, 2025
0a8d00c
Removed references to openshift-acct-mgt
QuanMPhm Aug 11, 2025
8d01d53
Merge pull request #234 from nerc-project/dependabot/github_actions/a…
knikolla Aug 27, 2025
e79cc36
Merge pull request #202 from QuanMPhm/feature/add_attr
knikolla Aug 27, 2025
c35be23
Merge pull request #233 from QuanMPhm/feature/clean_up_acct_manager
knikolla Aug 27, 2025
f4529a1
Created new allocation attribute for IBM storage quota
QuanMPhm Aug 6, 2025
eb5298d
Migrate to use NESE Storage GB Rate
KelvinLinBU Aug 27, 2025
eb4ab58
Merge pull request #232 from QuanMPhm/231/ibm_storage_attr
knikolla Aug 28, 2025
0ab7910
Merge pull request #240 from KelvinLinBU/migrate_nese
knikolla Aug 28, 2025
a8ff059
Include IBM Scale Storage in OpenShift storage calculations
QuanMPhm Aug 21, 2025
d08dbd9
Merge pull request #238 from QuanMPhm/237/ibm_in_invoice
knikolla Aug 28, 2025
85f2510
This lowers the minimum resources that a container must run with.
naved001 Sep 2, 2025
6a88cbb
Merge pull request #241 from naved001/openshift/lower-minimum-resources
knikolla Sep 8, 2025
f020245
Bump actions/setup-python from 5 to 6
dependabot[bot] Sep 8, 2025
1b2338d
Merge pull request #242 from nerc-project/dependabot/github_actions/a…
jtriley Sep 9, 2025
8d892d9
Allow validation of allocations with status `Active (Needs Renewal)`
QuanMPhm Sep 11, 2025
1cb7589
Merge pull request #247 from QuanMPhm/246/storage_renewal
knikolla Oct 3, 2025
6f61e46
Allow pushing user-allocation membership to Keycloak
QuanMPhm Oct 3, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 10 additions & 0 deletions .github/dependabot.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# Set update schedule for GitHub Actions

version: 2
updates:

- package-ecosystem: "github-actions"
directory: "/"
schedule:
# Check for updates to GitHub Actions every week
interval: "weekly"
22 changes: 22 additions & 0 deletions .github/workflows/pre-commit.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
name: pre-commit

on:
pull_request:
push:
branches: [main]

jobs:
pre-commit:
runs-on: ubuntu-latest

steps:
- name: Checkout code
uses: actions/checkout@v5

- name: Set up Python
uses: actions/setup-python@v6
with:
python-version: "3.12"

- name: Run pre-commit hooks
uses: pre-commit/[email protected]
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: test-py39-functional-microshift
name: test-functional-microshift

on:
push:
Expand All @@ -9,19 +9,19 @@ on:
env:
PYTHONWARNINGS: ignore
KUBECONFIG: ${{ github.workspace }}/kubeconfig
ACCT_MGT_VERSION: "6012025c247ab25fb2cab3be9ad06080e28713ee"
ACCT_MGT_VERSION: "cecbe131843e5033ab6f6fdce2b5a8af58d80f9d"

jobs:
build:
runs-on: ubuntu-20.04
runs-on: ubuntu-22.04

steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v5

- name: Set up Python 3.9
uses: actions/setup-python@v4
- name: Set up Python
uses: actions/setup-python@v6
with:
python-version: "3.9"
python-version: 3.12

- name: Fix for "grup-efi-amd64-signed" missing
run: |
Expand All @@ -35,6 +35,10 @@ jobs:
run: |
bash ./ci/setup-oc-client.sh

- name: Install Keycloak
run: |
bash ./ci/setup-keycloak.sh

- name: Install Microshift
run: |
./ci/microshift.sh
Expand Down
41 changes: 41 additions & 0 deletions .github/workflows/test-functional-microstack.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
name: test-functional-microstack

on:
push:
branches: [ main ]
pull_request:
branches: [ main ]

jobs:
build:
runs-on: ubuntu-22.04

steps:
- uses: actions/checkout@v5

- name: Set up Python
uses: actions/setup-python@v6
with:
python-version: 3.12

- name: Install Keycloak
run: |
bash ./ci/setup-keycloak.sh

- name: Install ColdFront and plugin
run: |
python -m pip install --upgrade pip
pip install -r test-requirements.txt
pip install -e .

- name: Install and start Ceph RadosGW
run: |
sudo ./ci/radosgw.sh

- name: Install and start Microstack
run: |
./ci/microstack.sh

- name: Run functional tests
run: |
./ci/run_functional_tests_openstack.sh
26 changes: 0 additions & 26 deletions .github/workflows/test-py38-functional-devstack.yaml

This file was deleted.

47 changes: 0 additions & 47 deletions .github/workflows/test-py39-functional.yaml

This file was deleted.

Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: test-py39-unit
name: test-unit

on:
push:
Expand All @@ -8,20 +8,20 @@ on:

jobs:
build:
runs-on: ubuntu-latest
runs-on: ubuntu-22.04

steps:
- uses: actions/checkout@v2
- uses: actions/checkout@v5

- name: Set up Python 3.9
uses: actions/setup-python@v1
- name: Set up Python
uses: actions/setup-python@v6
with:
python-version: 3.9
python-version: 3.12

- name: Install ColdFront and plugin
run: |
./ci/setup.sh

- name: Run functional tests
- name: Run unit tests
run: |
./ci/run_unit_tests.sh
17 changes: 17 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.5.0
hooks:
- id: trailing-whitespace
- id: end-of-file-fixer
- id: check-json
- id: check-merge-conflict
- id: check-case-conflict
- id: detect-private-key

- repo: https://github.com/astral-sh/ruff-pre-commit
rev: v0.11.6
hooks:
- id: ruff
args: [--fix]
- id: ruff-format
41 changes: 27 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,37 +78,36 @@ dashboard or through the helper command:

```bash
$ coldfront add_openstack_resource
usage: coldfront add_openstack_resource [-h] --name NAME --auth-url AUTH_URL [--users-domain USERS_DOMAIN] [--projects-domain PROJECTS_DOMAIN] --idp IDP
[--protocol PROTOCOL] [--role ROLE] [--version] [-v {0,1,2,3}] [--settings SETTINGS] [--pythonpath PYTHONPATH] [--traceback]
[--no-color] [--force-color]
usage: coldfront add_openstack_resource [-h] --name NAME --auth-url AUTH_URL [--users-domain USERS_DOMAIN] [--projects-domain PROJECTS_DOMAIN] --idp IDP [--protocol PROTOCOL] [--role ROLE]
[--public-network PUBLIC_NETWORK] [--network-cidr NETWORK_CIDR] [--esi] [--version] [-v {0,1,2,3}] [--settings SETTINGS] [--pythonpath PYTHONPATH] [--traceback]
[--no-color] [--force-color] [--skip-checks]
coldfront add_openstack_resource: error: the following arguments are required: --name, --auth-url, --idp
```

### Configuring for OpenShift
An Openstack resource can be specified as an ESI resource by setting the `--esi` command flag.
ESI resource allocations will only have quotas for network resources by default.

Note: OpenShift support requires deploying the [openshift-acct-mgt][]
API service.

[openshift-acct-mgt]: https://github.com/cci-moc/openshift-acct-mgt
### Configuring for OpenShift

Authentication for OpenShift is loaded as pairs of environment variables
`OPENSHIFT_{resource_name}_USERNAME` and `OPENSHIFT_{resource_name}_PASSWORD`
Authentication for OpenShift is loaded as a environment variable
`OPENSHIFT_{resource_name}_TOKEN` which should be a access token with appropriate permissions
where `{resource_name}` is the name of the coldfront resource as all uppercase
(with spaces and `-` replaced by `_`).

Each OpenShift resource must have the following attributes set in coldfront:
* `OpenStack Auth URL` - the URL of the `openshift-acct-mgt` endpoint.
* `OpenStack Role for User in Project` - the name of the `ClusterRole` to assign to users
* `OpenShift API URL` - the URL of the Openshift cluster API.
* `OpenShift Role for User in Project` - the name of the `ClusterRole` to assign to users
on the namespace.
* `OpenShift Identity Provider Name` - the name of the IDP configured in Openshift

Registration of OpenShift coldfront resources can be performed via the UI management
dashboard or through the helper command:

```bash
$ coldfront add_openshift_resource
usage: coldfront add_openshift_resource [-h] --name NAME --auth-url AUTH_URL [--role ROLE] [--version] [-v {0,1,2,3}] [--settings SETTINGS] [--pythonpath PYTHONPATH] [--traceback]
usage: coldfront add_openshift_resource [-h] --name NAME --api-url API_URL --idp IDP [--role ROLE] [--for-virtualization] [--version] [-v {0,1,2,3}] [--settings SETTINGS] [--pythonpath PYTHONPATH] [--traceback]
[--no-color] [--force-color] [--skip-checks]
coldfront add_openshift_resource: error: the following arguments are required: --name, --auth-url
coldfront add_openshift_resource: error: the following arguments are required: --name, --api-url, --idp
```

### Quotas
Expand Down Expand Up @@ -150,3 +149,17 @@ following resource allocation attribute types.

By submitting a Resource Allocation Change Request and editing those attributes
a PI can request a change in their quota.

## Pre-commit hooks
```
pip install pre-commit
```
Pre-commit runs tools like:
- [Ruff](https://docs.astral.sh/ruff/) — fast linter and fixer
- Basic checks like trailing whitespace removal, JSON validation, and more.

To set up Git hook locally:
```
pre-commit install
```
After this, every time you make a commit, the hooks will run automatically!
2 changes: 2 additions & 0 deletions ci/devstack-config-ip.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
HOST_IP=`ip addr show eth0 | grep "inet " | awk '{ print $2 }' | awk -F "/" '{ print $1 }'`
HOST_IPV6=`ip addr show eth0 | grep "inet6 " | awk '{ print $2 }' | awk -F "/" '{ print $1 }'`
60 changes: 0 additions & 60 deletions ci/devstack.sh

This file was deleted.

2 changes: 2 additions & 0 deletions ci/microshift.sh
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
#!/bin/bash

#
# Installs Microshift on Docker
#
Expand Down
Loading