Skip to content

Commit 345b5b0

Browse files
kamangirkamangir
authored
Merge pull request #16 from kamangir/literature-review-refactors-2024-08-07-86622
literature review refactors
2 parents 8607994 + 79f4b01 commit 345b5b0

33 files changed

+630
-306
lines changed

‎README.md

+6-50
Original file line numberDiff line numberDiff line change
@@ -6,59 +6,15 @@
66
pip install notebooks-and-scripts
77
```
88

9-
🔷 [scripts](#scripts) 🔷 [workflow](#workflow) 🔷
10-
11-
---
12-
13-
## scripts
14-
15-
scripts are maintained as legacy and are replaced with `@docker eval - <command>`.
16-
17-
| ![image](https://github.com/kamangir/assets/blob/main/nbs/3x4.jpg?raw=true) | ![image](https://github.com/kamangir/assets/blob/main/nbs/mission-patch-00008.png?raw=true) | ![image](https://github.com/kamangir/assets/blob/main/nbs/train-summary.png?raw=true) ![image](https://github.com/kamangir/assets/blob/main/nbs/predict-00000.png?raw=true) |
18-
| --------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
19-
| [paint-a-sentence](./scripts/paint-a-sentence.sh) | [mission-patch](./scripts/mission-patch.sh) | [roofAI-train](./scripts/roofAI-train.sh) |
20-
| https://medium.com/@arash-kamangir/a-cat-walking-under-apple-trees-style-by-stable-diffusion-ab60ece43e2a | https://medium.com/@arash-kamangir/private-mission-patch-%EF%B8%8F-1-0c2eddd79762 | https://arash-kamangir.medium.com/roofai-%EF%B8%8F-on-gpu-6-b02f8f67ed3f |
21-
22-
also: [`sagesemseg`](./scripts/sagesemseg/).
23-
24-
---
25-
26-
## workflow
27-
28-
also home to [`workflow`](./notebooks_and_scripts/workflow/generic.py), an abstraction to run mixed-type (cpu/gpu/...) [DAG](https://networkx.org/documentation/stable/reference/classes/digraph.html)s of bash commands on aws batch.
29-
30-
```bash
31-
> workflow help
32-
workflow create \
33-
pattern=a-bc-d|hourglass,~upload \
34-
-|<job-name> \
35-
<command-line>
36-
. create a <pattern> workflow.
37-
workflow monitor \
38-
~download,~upload \
39-
.|<job-name>
40-
. monitor workflow.
41-
workflow submit \
42-
~download,dryrun,to=aws_batch|generic|local|localflow,~upload \
43-
.|<job-name>
44-
. submit workflow.
45-
```
46-
47-
```bash
48-
@select - open
49-
workflow create pattern=hourglass .
50-
workflow submit to=aws_batch
51-
@watch - @download
52-
```
53-
54-
from https://arash-kamangir.medium.com/%EF%B8%8F-openai-experiments-54-e49117dc69ef
55-
56-
| [`a-bc-d`](./notebooks_and_scripts/workflow/patterns/a-bc-d.dot) [🔗](https://kamangir-public.s3.ca-central-1.amazonaws.com/a-bc-d/workflow.gif?raw=true) | [`hourglass`](./notebooks_and_scripts/workflow/patterns/hourglass.dot) [🔗](https://kamangir-public.s3.ca-central-1.amazonaws.com/hourglass/workflow.gif?raw=true) | [`map-reduce`](./notebooks_and_scripts/workflow/patterns/map-reduce.dot) [🔗](https://kamangir-public.s3.ca-central-1.amazonaws.com/map-reduce/workflow.gif?raw=true) |
57-
| --------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
58-
| ![image](https://kamangir-public.s3.ca-central-1.amazonaws.com/a-bc-d/workflow.gif?raw=true) | ![image](https://kamangir-public.s3.ca-central-1.amazonaws.com/hourglass/workflow.gif?raw=true) | ![image](https://kamangir-public.s3.ca-central-1.amazonaws.com/map-reduce/workflow.gif?raw=true) |
9+
| | |
10+
| --- | --- |
11+
| 📜[`workflow`](https://github.com/kamangir/notebooks-and-scripts/tree/main/notebooks_and_scripts/workflow) [![image](https://kamangir-public.s3.ca-central-1.amazonaws.com/hourglass/workflow.gif?raw=true)](https://github.com/kamangir/notebooks-and-scripts/tree/main/notebooks_and_scripts/workflow) an abstraction to run mixed-type (cpu/gpu/...) [DAG](https://networkx.org/documentation/stable/reference/classes/digraph.html)s of commands on [aws batch](https://aws.amazon.com/batch/). | 📜[`scripts`](https://github.com/kamangir/notebooks-and-scripts/tree/main/scripts) [![image](https://github.com/kamangir/assets/blob/main/nbs/3x4.jpg?raw=true)](https://github.com/kamangir/notebooks-and-scripts/tree/main/scripts) legacy mechanisms replaced with `@docker eval - <command>` and `@batch eval - <command>`. |
5912

6013
---
6114

6215
[![PyPI version](https://img.shields.io/pypi/v/notebooks-and-scripts.svg)](https://pypi.org/project/notebooks-and-scripts/)
6316

6417
To use on [AWS SageMaker](https://aws.amazon.com/sagemaker/) replace `<plugin-name>` with `nbs` and follow [these instructions](https://github.com/kamangir/notebooks-and-scripts/blob/main/SageMaker.md).
18+
19+
---
20+
built by [`abcli-9.192.1-current`](https://github.com/kamangir/awesome-bash-cli), based on [`notebooks_and_scripts-4.642.1`](https://github.com/kamangir/notebooks-and-scripts).

‎blueness/pip.ipynb ‎notebooks/blueness/pip.ipynb

File renamed without changes.

‎notebooks_and_scripts/.abcli/actions.sh

+2
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
#! /usr/bin/env bash
22

33
function notebooks_and_scripts_action_git_before_push() {
4+
notebooks_and_scripts build_README
5+
46
[[ "$(abcli_git get_branch)" == "main" ]] &&
57
notebooks_and_scripts pypi build
68
}

‎notebooks_and_scripts/.abcli/tests/README.sh

+8
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
#! /usr/bin/env bash
2+
3+
function openai_commands_plugin_README() {
4+
local options=$1
5+
6+
abcli_eval ,$options \
7+
openai_commands build_README
8+
}

‎notebooks_and_scripts/.abcli/tests/workflow_runner.sh

+22-7
Original file line numberDiff line numberDiff line change
@@ -3,19 +3,34 @@
33
function test_notebooks_and_scripts_worflow_runner() {
44
local options=$1
55
local do_dryrun=$(abcli_option_int "$options" dryrun 0)
6+
local list_of_runners=$(abcli_option "$options" runner $NBS_RUNNERS_LIST)
7+
local list_of_patterns=$(abcli_option "$options" pattern $NBS_PATTRENS_LIST)
68

79
local pattern
8-
for pattern in $(echo $NBS_PATTRENS_LIST | tr \| " "); do
9-
abcli_log "testing pattern=$pattern"
10+
local runner
11+
for runner in $(echo $list_of_runners | tr \| " "); do
12+
for pattern in $(echo $list_of_patterns | tr \| " "); do
13+
abcli_log "📜 testing runner=$runner, pattern=$pattern ..."
1014

11-
local job_name=$pattern-$(abcli_string_timestamp)
15+
local job_name=$runner-$pattern-$(abcli_string_timestamp)
1216

13-
workflow create pattern=$pattern $job_name
17+
workflow create \
18+
pattern=$pattern \
19+
$job_name \
20+
--publish_as $runner-$pattern
21+
[[ $? -ne 0 ]] && return 1
1422

15-
workflow submit ~download,to=aws_batch $job_name
23+
workflow submit \
24+
to=$runner \
25+
$job_name
26+
[[ $? -ne 0 ]] && return 1
1627

17-
workflow monitor ~download $job_name
28+
workflow monitor \
29+
publish_as=$runner-$pattern \
30+
$job_name
31+
[[ $? -ne 0 ]] && return 1
1832

19-
abcli_hr
33+
abcli_hr
34+
done
2035
done
2136
}

‎notebooks_and_scripts/.abcli/workflow/config.sh

+3-16
Original file line numberDiff line numberDiff line change
@@ -1,20 +1,7 @@
11
#! /usr/bin/env bash
22

3-
function notebooks_and_scripts_workflow_config() {
4-
local options=$1
5-
local use_cache=$(abcli_option_int "$options" cache 1)
3+
export NBS_RUNNERS_LIST=$(python3 -m notebooks_and_scripts.workflow.runners list --delim \|)
64

7-
export NBS_RUNNERS_LIST="aws_batch|generic|local|localflow"
8-
[[ "$use_cache" == 0 ]] &&
9-
export NBS_RUNNERS_LIST=$(python3 -m notebooks_and_scripts.workflow.runners list --delim \|)
5+
export NBS_PATTRENS_LIST=$(python3 -m notebooks_and_scripts.workflow.patterns list --delim \|)
106

11-
export NBS_PATTRENS_LIST="a-bc-d|hourglass|map-reduce"
12-
[[ "$use_cache" == 0 ]] &&
13-
export NBS_PATTRENS_LIST=$(python3 -m notebooks_and_scripts.workflow.patterns list --delim \|)
14-
15-
export NBS_PATTREN_DEFAULT="a-bc-d"
16-
[[ "$use_cache" == 0 ]] &&
17-
export NBS_PATTREN_DEFAULT=$(python3 -m notebooks_and_scripts.workflow.patterns list --count 1)
18-
}
19-
20-
notebooks_and_scripts_workflow_config "$@"
7+
export NBS_PATTREN_DEFAULT=$(python3 -m notebooks_and_scripts.workflow.patterns list --count 1)

‎notebooks_and_scripts/.abcli/workflow/create.sh

+5-3
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,9 @@ function notebooks_and_scripts_workflow_create() {
44
local options=$1
55

66
if [ $(abcli_option_int "$options" help 0) == 1 ]; then
7+
local args="[--publish_as <public-object-name>]"
78
options="pattern=$NBS_PATTRENS_LIST$EOP,~upload"
8-
abcli_show_usage "workflow create$ABCUL$options$ABCUL.|<job-name>$EOPE" \
9+
abcli_show_usage "workflow create$ABCUL$options$ABCUL.|<job-name>$ABCUL$args$EOPE" \
910
"create a <pattern> workflow."
1011
return
1112
fi
@@ -15,12 +16,13 @@ function notebooks_and_scripts_workflow_create() {
1516

1617
local job_name=$(abcli_clarify_object $2 .)
1718

18-
abcli_log "creating workflow: $pattern -> $job_name"
19+
abcli_log "📜 workflow.create: $pattern -> $job_name"
1920

2021
python3 -m notebooks_and_scripts.workflow \
2122
create \
2223
--job_name $job_name \
23-
--pattern "$pattern"
24+
--pattern "$pattern" \
25+
"${@:3}"
2426

2527
[[ "$do_upload" == 1 ]] &&
2628
abcli_upload - $job_name

‎notebooks_and_scripts/.abcli/workflow/monitor.sh

+6-8
Original file line numberDiff line numberDiff line change
@@ -4,26 +4,24 @@ function notebooks_and_scripts_workflow_monitor() {
44
local options=$1
55

66
if [ $(abcli_option_int "$options" help 0) == 1 ]; then
7-
local options="$EOP~download,node=<node>,publish,~upload"
7+
local options="$EOP~download,node=<node>,publish_as=<public-object-name>,~upload"
88
abcli_show_usage "workflow monitor$ABCUL$options$ABCUL.|<job-name>$EOPE" \
99
"monitor workflow."
1010
return
1111
fi
1212

1313
local do_download=$(abcli_option_int "$options" download 1)
14-
local do_publish=$(abcli_option_int "$options" publish 0)
14+
local publish_as=$(abcli_option "$options" publish_as)
1515
local do_upload=$(abcli_option_int "$options" upload 1)
1616
local node=$(abcli_option "$options" node void)
1717

1818
local job_name=$(abcli_clarify_object $2 .)
1919

20+
abcli_log "📜 workflow.monitor: $job_name @ $node ..."
21+
2022
[[ "$do_download" == 1 ]] &&
2123
abcli_download - $job_name
2224

23-
local pattern=$(abcli_metadata get \
24-
key=load_pattern.pattern,object \
25-
$job_name)
26-
2725
python3 -m notebooks_and_scripts.workflow.runners \
2826
monitor \
2927
--hot_node $node \
@@ -32,8 +30,8 @@ function notebooks_and_scripts_workflow_monitor() {
3230
[[ "$do_upload" == 1 ]] &&
3331
abcli_upload - $job_name
3432

35-
[[ "$do_publish" == 1 ]] &&
36-
abcli_publish as=$pattern,~download,suffix=.gif $job_name
33+
[[ ! -z "$publish_as" ]] &&
34+
abcli_publish as=$publish_as,~download,suffix=.gif $job_name
3735

3836
return 0
3937
}

‎notebooks_and_scripts/.abcli/workflow/submit.sh

+9-1
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,7 @@ function notebooks_and_scripts_workflow_submit() {
2020
[[ "$do_download" == 1 ]] &&
2121
abcli_download - $job_name
2222

23-
abcli_log "workflow submit: $job_name -> $runner_type"
23+
abcli_log "📜 workflow.submit: $job_name -> $runner_type"
2424

2525
python3 -m notebooks_and_scripts.workflow.runners \
2626
submit \
@@ -32,5 +32,13 @@ function notebooks_and_scripts_workflow_submit() {
3232
[[ "$do_upload" == 1 ]] &&
3333
abcli_upload - $job_name
3434

35+
[[ "$status" -ne 0 ]] && return $status
36+
37+
if [[ "$runner_type" == local ]]; then
38+
abcli_eval dryrun=$do_dryrun \
39+
source $abcli_object_root/$job_name/$job_name.sh
40+
status="$?"
41+
fi
42+
3543
return $status
3644
}

‎notebooks_and_scripts/README.py

+72
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
import os
2+
from abcli import file
3+
from abcli.plugins.README import build as build_README
4+
from notebooks_and_scripts import NAME, VERSION, ICON, REPO_NAME
5+
from notebooks_and_scripts.workflow.README import items as workflow_items
6+
7+
8+
features = {
9+
"workflow": {
10+
"description": "an abstraction to run mixed-type (cpu/gpu/...) [DAG](https://networkx.org/documentation/stable/reference/classes/digraph.html)s of commands on [aws batch](https://aws.amazon.com/batch/).",
11+
"icon": ICON,
12+
"thumbnail": "https://kamangir-public.s3.ca-central-1.amazonaws.com/hourglass/workflow.gif?raw=true",
13+
"url": "https://github.com/kamangir/notebooks-and-scripts/tree/main/notebooks_and_scripts/workflow",
14+
},
15+
"scripts": {
16+
"description": "legacy mechanisms replaced with `@docker eval - <command>` and `@batch eval - <command>`.",
17+
"icon": ICON,
18+
"thumbnail": "https://github.com/kamangir/assets/blob/main/nbs/3x4.jpg?raw=true",
19+
"url": "https://github.com/kamangir/notebooks-and-scripts/tree/main/scripts",
20+
},
21+
"template": {
22+
"description": "",
23+
"icon": "",
24+
"thumbnail": "",
25+
"url": "",
26+
},
27+
}
28+
29+
items = [
30+
"{}[`{}`]({}) [![image]({})]({}) {}".format(
31+
details["icon"],
32+
feature,
33+
details["url"],
34+
details["thumbnail"],
35+
details["url"],
36+
details["description"],
37+
)
38+
for feature, details in features.items()
39+
if feature != "template"
40+
]
41+
42+
43+
def build():
44+
return build_README(
45+
items=items,
46+
cols=2,
47+
template_filename=os.path.join(
48+
file.path(__file__),
49+
"./assets/README.md",
50+
),
51+
filename=os.path.join(
52+
file.path(__file__),
53+
"../README.md",
54+
),
55+
NAME=NAME,
56+
VERSION=VERSION,
57+
REPO_NAME=REPO_NAME,
58+
) and build_README(
59+
items=workflow_items,
60+
cols=4,
61+
template_filename=os.path.join(
62+
file.path(__file__),
63+
"./assets/workflow.md",
64+
),
65+
filename=os.path.join(
66+
file.path(__file__),
67+
"./workflow/README.md",
68+
),
69+
NAME=NAME,
70+
VERSION=VERSION,
71+
REPO_NAME=REPO_NAME,
72+
)

‎notebooks_and_scripts/__init__.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44

55
DESCRIPTION = f"{ICON} notebooks and scripts for ai experiments and aws batch jobs."
66

7-
VERSION = "4.598.1"
7+
VERSION = "4.642.1"
88

99
REPO_NAME = "notebooks-and-scripts"
1010

‎notebooks_and_scripts/__main__.py

+11-1
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,17 @@
11
from notebooks_and_scripts import NAME, VERSION, DESCRIPTION, ICON
22
from notebooks_and_scripts.logger import logger
3+
from notebooks_and_scripts import README
34
from blueness.argparse.generic import main
45

5-
success, message = main(__file__, NAME, VERSION, DESCRIPTION, ICON)
6+
success, message = main(
7+
__file__,
8+
NAME,
9+
VERSION,
10+
DESCRIPTION,
11+
ICON,
12+
{
13+
"build_README": lambda _: README.build(),
14+
},
15+
)
616
if not success:
717
logger.error(message)

‎notebooks_and_scripts/assets/README.md

+17
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
# 📜 notebooks & scripts
2+
3+
📜 [notebooks](./notebooks) and [scripts](./scripts) for ai experiments and aws batch jobs.
4+
5+
```bash
6+
pip install notebooks-and-scripts
7+
```
8+
9+
--table--
10+
11+
---
12+
13+
[![PyPI version](https://img.shields.io/pypi/v/notebooks-and-scripts.svg)](https://pypi.org/project/notebooks-and-scripts/)
14+
15+
To use on [AWS SageMaker](https://aws.amazon.com/sagemaker/) replace `<plugin-name>` with `nbs` and follow [these instructions](https://github.com/kamangir/notebooks-and-scripts/blob/main/SageMaker.md).
16+
17+
--signature--

‎notebooks_and_scripts/assets/workflow.md

+29
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
# 📜 workflow
2+
3+
an abstraction to run mixed-type (cpu/gpu/...) [DAG](https://networkx.org/documentation/stable/reference/classes/digraph.html)s of commands with dependencies on [aws batch](https://aws.amazon.com/batch/), and a few other compute resources.
4+
5+
```bash
6+
> workflow help
7+
workflow create \
8+
pattern=a-bc-d|hourglass|map-reduce,~upload \
9+
.|<job-name> \
10+
[--publish_as <public-object-name>]
11+
. create a <pattern> workflow.
12+
workflow monitor \
13+
~download,node=<node>,publish_as=<public-object-name>,~upload \
14+
.|<job-name>
15+
. monitor workflow.
16+
workflow submit \
17+
~download,dryrun,to=aws_batch|generic|local,~upload \
18+
.|<job-name>
19+
. submit workflow.
20+
```
21+
22+
example use: [literature review using OpenAI API](https://github.com/kamangir/openai-commands/tree/main/openai_commands/literature_review).
23+
24+
--table--
25+
26+
27+
related: [1](https://arash-kamangir.medium.com/%EF%B8%8F-openai-experiments-54-e49117dc69ef)
28+
29+
--signature--

‎notebooks_and_scripts/aws_batch/__init__.py

-3
Original file line numberDiff line numberDiff line change
@@ -1,3 +0,0 @@
1-
from notebooks_and_scripts import NAME, VERSION
2-
3-
NAME = f"{NAME}.aws_batch"

0 commit comments

Comments
 (0)