Skip to content

Commit 719df76

Browse files
ghukillghukill
authored
Merge pull request #363 from MITLibraries/TIMX-491-remove-parquet-feature-flags
TIMX 491 - remove parquet feature flags
2 parents 370f5b7 + 280262f commit 719df76

34 files changed

+1593
-2585
lines changed

‎.pre-commit-config.yaml

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -22,8 +22,8 @@ repos:
2222
language: system
2323
pass_filenames: true
2424
types: ["python"]
25-
- id: safety
26-
name: safety
27-
entry: pipenv check
25+
- id: pip-audit
26+
name: pip-audit
27+
entry: pipenv run pip-audit
2828
language: system
2929
pass_filenames: false

‎Makefile

Lines changed: 2 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -49,11 +49,10 @@ ruff: # Run 'ruff' linter and print a preview of errors
4949
pipenv run ruff check .
5050

5151
safety: # Check for security vulnerabilities and verify Pipfile.lock is up-to-date
52-
pipenv check
52+
pipenv run pip-audit
5353
pipenv verify
5454

55-
lint-apply: # Apply changes with 'black' and resolve 'fixable errors' with 'ruff'
56-
black-apply ruff-apply
55+
lint-apply: black-apply ruff-apply # Apply changes with 'black' and resolve 'fixable errors' with 'ruff'
5756

5857
black-apply: # Apply changes with 'black'
5958
pipenv run black .

‎Pipfile

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,8 @@ pre-commit = "*"
2323
pytest = "*"
2424
ruff = "*"
2525
vcrpy = "*"
26+
pip-audit = "*"
27+
ipython = "*"
2628

2729
[requires]
2830
python_version = "3.12"

‎Pipfile.lock

Lines changed: 996 additions & 615 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎README.md

Lines changed: 26 additions & 28 deletions
Original file line numberDiff line numberDiff line change
@@ -115,33 +115,31 @@ SENTRY_DSN=### If set to a valid Sentry DSN, enables Sentry exception monitoring
115115
All CLI commands can be run with `pipenv run`.
116116
117117
```
118-
Usage: tim [OPTIONS] COMMAND [ARGS]...
119-
120-
TIM provides commands for interacting with OpenSearch indexes.
121-
For more details on a specific command, run tim COMMAND -h.
122-
123-
╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
124-
│ --url -u TEXT The OpenSearch instance endpoint minus the http scheme, e.g. │
125-
│ 'search-timdex-env-1234567890.us-east-1.es.amazonaws.com'. If not provided, will attempt to get from the │
126-
│ TIMDEX_OPENSEARCH_ENDPOINT environment variable. Defaults to 'localhost'. │
127-
│ --verbose -v Pass to log at debug level instead of info │
128-
│ --help -h Show this message and exit. │
129-
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
130-
╭─ Get cluster-level information ─────────────────────────────────────────────────────────────────────────────────────────────────────╮
131-
│ ping Ping OpenSearch and display information about the cluster. │
132-
│ indexes Display summary information about all indexes in the cluster. │
133-
│ aliases List OpenSearch aliases and their associated indexes. │
134-
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
135-
╭─ Index management commands ─────────────────────────────────────────────────────────────────────────────────────────────────────────╮
136-
│ create Create a new index in the cluster. │
137-
│ delete Delete an index. │
138-
│ promote Promote index as the primary alias and add it to any additional provided aliases. │
139-
│ demote Demote an index from all its associated aliases. │
140-
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
141-
╭─ Bulk record processing commands ───────────────────────────────────────────────────────────────────────────────────────────────────╮
142-
│ bulk-index Bulk index records into an index. │
143-
│ bulk-delete Bulk delete records from an index. │
144-
│ bulk-update Bulk update records from an index. │
145-
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
118+
Usage: tim [OPTIONS] COMMAND [ARGS]...
119+
120+
TIM provides commands for interacting with OpenSearch indexes.
121+
For more details on a specific command, run tim COMMAND -h.
122+
123+
╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
124+
│ --url -u TEXT The OpenSearch instance endpoint minus the http scheme, e.g. │
125+
│ 'search-timdex-env-1234567890.us-east-1.es.amazonaws.com'. If not provided, will attempt to get from the │
126+
│ TIMDEX_OPENSEARCH_ENDPOINT environment variable. Defaults to 'localhost'. │
127+
│ --verbose -v Pass to log at debug level instead of info │
128+
│ --help -h Show this message and exit. │
129+
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
130+
╭─ Get cluster-level information ────────────────────────────────────────────────────────────────────────────────────────────────╮
131+
│ ping Ping OpenSearch and display information about the cluster. │
132+
│ indexes Display summary information about all indexes in the cluster. │
133+
│ aliases List OpenSearch aliases and their associated indexes. │
134+
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
135+
╭─ Index management commands ────────────────────────────────────────────────────────────────────────────────────────────────────╮
136+
│ create Create a new index in the cluster. │
137+
│ delete Delete an index. │
138+
│ promote Promote index as the primary alias and add it to any additional provided aliases. │
139+
│ demote Demote an index from all its associated aliases. │
140+
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
141+
╭─ Bulk record processing commands ──────────────────────────────────────────────────────────────────────────────────────────────╮
142+
│ bulk-update Bulk update records for an index. │
143+
╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
146144
```
147145

‎tests/conftest.py

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
import pytest
22
import vcr
33
from click.testing import CliRunner
4+
from timdex_dataset_api import TIMDEXDataset
45

56
import tim.opensearch as tim_os
67

@@ -33,3 +34,38 @@ def test_opensearch_client():
3334
@pytest.fixture
3435
def runner():
3536
return CliRunner()
37+
38+
39+
@pytest.fixture
40+
def timdex_dataset() -> TIMDEXDataset:
41+
td = TIMDEXDataset("tests/fixtures/dataset")
42+
td.load()
43+
return td
44+
45+
46+
@pytest.fixture
47+
def five_valid_index_libguides_records(timdex_dataset):
48+
return timdex_dataset.read_transformed_records_iter(
49+
run_id="85cfe316-089c-4639-a5af-c861a7321493"
50+
)
51+
52+
53+
@pytest.fixture
54+
def one_invalid_index_libguides_records(timdex_dataset):
55+
return timdex_dataset.read_transformed_records_iter(
56+
run_id="21e7f272-7b96-480c-9c25-36075355fc4c"
57+
)
58+
59+
60+
@pytest.fixture
61+
def one_valid_delete_libguides_records(timdex_dataset):
62+
return timdex_dataset.read_transformed_records_iter(
63+
run_id="59d938b9-df61-481b-bec9-9d9eb8fbf21c"
64+
)
65+
66+
67+
@pytest.fixture
68+
def one_valid_delete_libguides_records_not_found(timdex_dataset):
69+
return timdex_dataset.read_transformed_records_iter(
70+
run_id="3718935d-2bc0-4385-919e-c7a83238215e"
71+
)

‎tests/fixtures/cassettes/cli/bulk_delete_with_index_name_success.yaml

Lines changed: 0 additions & 72 deletions
This file was deleted.

‎tests/fixtures/cassettes/cli/bulk_delete_with_source_success.yaml

Lines changed: 0 additions & 72 deletions
This file was deleted.

‎tests/fixtures/cassettes/opensearch/TO_DELETE/bulk_delete_record_not_found.yaml

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
interactions:
2+
- request:
3+
body: '{"delete":{"_id":"no-record-here","_index":"alma-2022-09-01t00-00-00"}}
4+
5+
'
6+
headers:
7+
Content-Length:
8+
- '72'
9+
content-type:
10+
- application/json
11+
user-agent:
12+
- opensearch-py/2.0.1 (Python 3.10.8)
13+
method: POST
14+
uri: http://localhost:9200/_bulk
15+
response:
16+
body:
17+
string: '{"took":5,"errors":false,"items":[{"delete":{"_index":"alma-2022-09-01t00-00-00","_type":"_doc","_id":"no-record-here","_version":1,"result":"not_found","_shards":{"total":2,"successful":1,"failed":0},"_seq_no":27,"_primary_term":1,"status":404}}]}'
18+
headers:
19+
content-length:
20+
- '248'
21+
content-type:
22+
- application/json; charset=UTF-8
23+
status:
24+
code: 200
25+
message: OK
26+
- request:
27+
body: null
28+
headers:
29+
Content-Length:
30+
- '0'
31+
content-type:
32+
- application/json
33+
user-agent:
34+
- opensearch-py/2.0.1 (Python 3.10.8)
35+
method: POST
36+
uri: http://localhost:9200/alma-2022-09-01t00-00-00/_refresh
37+
response:
38+
body:
39+
string: '{"_shards":{"total":2,"successful":1,"failed":0}}'
40+
headers:
41+
content-length:
42+
- '49'
43+
content-type:
44+
- application/json; charset=UTF-8
45+
status:
46+
code: 200
47+
message: OK
48+
version: 1

‎tests/fixtures/cassettes/opensearch/bulk_delete_record_weird_response.yaml renamed to ‎tests/fixtures/cassettes/opensearch/TO_DELETE/bulk_delete_record_weird_response.yaml

File renamed without changes.

0 commit comments

Comments
 (0)