Skip to content

Interactive generation script #53

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 18 commits into
base: master
Choose a base branch
from
Open

Interactive generation script #53

Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
8b2c770
first try gen script
younesbelkada Jun 10, 2022
62fcebb
add args
younesbelkada Jun 10, 2022
1782c2d
correct tokenizer
younesbelkada Jun 10, 2022
80df231
add import torch
younesbelkada Jun 10, 2022
0d11810
few nits
younesbelkada Jun 10, 2022
5899ed1
new line
younesbelkada Jun 10, 2022
609b880
print decoded output
younesbelkada Jun 10, 2022
877f76e
add small nit
younesbelkada Jun 10, 2022
560b3fc
fix small nit
younesbelkada Jun 10, 2022
5e2d2a2
forward contrib credits
younesbelkada Jun 10, 2022
7c6c171
rm useless file
younesbelkada Jun 10, 2022
d6dd676
small nits
younesbelkada Jun 10, 2022
21eca50
Update evaluation/generation/generate.py
younesbelkada Jun 27, 2022
839622c
Update evaluation/generation/generate.py
younesbelkada Jun 27, 2022
ac16dd7
commit suggestions
younesbelkada Jun 28, 2022
7191802
Update evaluation/generation/generate.py
younesbelkada Jun 28, 2022
5759518
fix small accelerate nits
younesbelkada Jul 8, 2022
e726b15
Merge branch 'master' into fix_generate
younesbelkada Jul 8, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions evaluation/generation/generate.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,8 @@ def get_args():
parser.add_argument("--greedy", action="store_true")
parser.add_argument("--top-k", type=int, default=0)
parser.add_argument("--offload_folder", type=str, help="offload folder for accelerate", default="./offload")
parser.add_argument("--max_memory", type=str, help="max memory per GPU", default="30GB")
parser.add_argument("--max_cpu_memory", type=str, help="max memory on CPU", default="300GB")

return parser.parse_args()

Expand Down Expand Up @@ -40,9 +42,12 @@ def main():
args.checkpoint,
device_map="auto" if args.parallelize else None,
torch_dtype=torch.bfloat16,
revision="gs{}".format(args.global_step) if args.global_step else None
offload_folder=args.offload_folder is args.parallelize else None,
revision="gs{}".format(args.global_step) if args.global_step else None,
max_memory=args.max_memory if args.parallelize else None,
max_cpu_memory=args.max_cpu_memory if args.parallelize else None,
offload_folder=args.offload_folder if args.parallelize else None,
)

print(f"Loaded model in {datetime.datetime.now() - start}")

text = ''
Expand Down