initial implementation

Author: dsyme

.dockerignore

@@ -0,0 +1,3 @@
+node_modules
+dist
+lib

.env.example

@@ -0,0 +1,20 @@
+{
+  "root": true,
+  "parser": "@typescript-eslint/parser",
+  "parserOptions": {
+    "project": "./tsconfig.json"
+  },
+  "plugins": ["@typescript-eslint"],
+  "extends": [
+    "eslint:recommended",
+    "plugin:@typescript-eslint/eslint-recommended",
+    "plugin:@typescript-eslint/recommended"
+  ],
+  "rules": {
+    "@typescript-eslint/no-floating-promises": "error",
+    "@typescript-eslint/no-misused-promises": "error",
+    "@typescript-eslint/no-explicit-any": 0,
+    "@typescript-eslint/no-unused-vars": 0,
+    "@typescript-eslint/ban-ts-comment": 0
+  }
+}

.eslintignore

@@ -1,7 +1,3 @@
 node_modules
-npm-debug.log
-*.pem
-!mock-cert.pem
-.env
-coverage
 lib
+.gpte

.eslintrc

@@ -0,0 +1,5 @@
+#!/usr/bin/env sh
+. "$(dirname -- "$0")/_/husky.sh"
+
+npx pretty-quick --staged
+

.gitignore

@@ -0,0 +1,4 @@
+{
+  "tabWidth": 2,
+  "printWidth": 100
+}

.husky/pre-commit

@@ -0,0 +1,4 @@
+{
+  "editor.defaultFormatter": "esbenp.prettier-vscode",
+  "editor.formatOnSave": true
+}

.prettierrc

@@ -0,0 +1,5 @@
+# For more information, see [docs](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/customizing-your-repository/about-code-owners#codeowners-syntax)
+
+# This repository is maintained by:
+
+- @dsyme @wunderalbert @johanrosenkilde

.vscode/settings.json

@@ -6,8 +6,8 @@ In the interest of fostering an open and welcoming environment, we as
 contributors and maintainers pledge to making participation in our project and
 our community a harassment-free experience for everyone, regardless of age, body
 size, disability, ethnicity, gender identity and expression, level of experience,
-education, socio-economic status, nationality, personal appearance, race,
-religion, or sexual identity and orientation.
+nationality, personal appearance, race, religion, or sexual identity and
+orientation.
 
 ## Our Standards
 
@@ -55,7 +55,7 @@ further defined and clarified by project maintainers.
 ## Enforcement
 
 Instances of abusive, harassing, or otherwise unacceptable behavior may be
-reported by contacting the project team at . All
+reported by contacting the project team at [email protected]. All
 complaints will be reviewed and investigated and will result in a response that
 is deemed necessary and appropriate to the circumstances. The project team is
 obligated to maintain confidentiality with regard to the reporter of an incident.
@@ -68,6 +68,7 @@ members of the project's leadership.
 ## Attribution
 
 This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
-available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
+available at [http://contributor-covenant.org/version/1/4][version]
 
-[homepage]: https://www.contributor-covenant.org
+[homepage]: http://contributor-covenant.org
+[version]: http://contributor-covenant.org/version/1/4/

CODEOWNERS

@@ -1,36 +1,10 @@
 ## Contributing
 
-[fork]: /fork
-[pr]: /compare
-[code-of-conduct]: CODE_OF_CONDUCT.md
-
 Hi there! We're thrilled that you'd like to contribute to this project. Your help is essential for keeping it great.
 
-Please note that this project is released with a [Contributor Code of Conduct][code-of-conduct]. By participating in this project you agree to abide by its terms.
-
-## Issues and PRs
-
-If you have suggestions for how this project could be improved, or want to report a bug, open an issue! We'd love all and any contributions. If you have questions, too, we'd love to hear them.
-
-We'd also love PRs. If you're thinking of a large PR, we advise opening up an issue first to talk about it, though! Look at the links below if you're not sure how to open a PR.
-
-## Submitting a pull request
-
-1. [Fork][fork] and clone the repository.
-1. Configure and install the dependencies: `npm install`.
-1. Make sure the tests pass on your machine: `npm test`, note: these tests also apply the linter, so there's no need to lint separately.
-1. Create a new branch: `git checkout -b my-branch-name`.
-1. Make your change, add tests, and make sure the tests still pass.
-1. Push to your fork and [submit a pull request][pr].
-1. Pat your self on the back and wait for your pull request to be reviewed and merged.
-
-Here are a few things you can do that will increase the likelihood of your pull request being accepted:
-
-- Write and update tests.
-- Keep your changes as focused as possible. If there are multiple changes you would like to make that are not dependent upon each other, consider submitting them as separate pull requests.
-- Write a [good commit message](http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html).
+This project is primarily for reference for the GitHub Next investigation it describes. Please contribute corrections.
 
-Work in Progress pull requests are also welcome to get feedback early on, or if there is something blocked you.
+Please note that this project is released with a [Contributor Code of Conduct](CODE_OF_CONDUCT.md). By participating in this project you agree to abide by its terms.
 
 ## Resources
 

CODE_OF_CONDUCT.md

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright GitHub
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

CONTRIBUTING.md

@@ -1,33 +1,68 @@
-# changepilot-bot
+# GPT-4 Equipped with Numeric Calculation
 
-> A GitHub App built with [Probot](https://github.com/probot/probot) that A bot that turns GitHub Issues into PRs
+GPT-4 has trouble with calculating with numbers. We look at a technique to fix that.
 
-## Setup
+[**Read the report - GPT-4 Equipped with Numeric Calculation**](docs/report.md)
+
+[**Read the evaluation**](docs/eval.md)
+
+> NOTE: The default model used when running this code is OpenAI's **`text-davinci-003`** and not GPT-4. The code can be adjusted to run against any completion API by changing the settings in [`src/engine/options.ts`](src/engine/options.ts) and [`src/engine/settings.ts`](src/engine/settings.ts).
+>
+> When writing this report, we evaluated the technique using the completion API of a private release of GPT-4. The public release of GPT-4 currently only provides a "chat" API. As a result some aspects of the technique described may need to be reworked because of this, hwoever we are providing the code and our investigation for reference. We believe the evaluation results remain valid although some variation in numbers is to be expected.
+
+## Requirements
 
 ```sh
-# Install dependencies
+sudo apt-get install npm
 npm install
+npm run build
+```
 
-# Run the bot
-npm start
+## Examples
+
+Define an API key:
+
+```bash
+export OPENAI_API_KEY=...
 ```
 
-## Docker
+Try your own question:
 
-```sh
-# 1. Build container
-docker build -t changepilot-bot .
+```bash
+./gpte ask --question "What is sin(13.31) where the input is in degrees?"
+./gpte ask --question "What is sin(13.31) where the input is in degrees?" --arith
 
-# 2. Start container
-docker run -e APP_ID=<app-id> -e PRIVATE_KEY=<pem-value> changepilot-bot
+./gpte ask --questionfile test/samples/number-years-grow-30.txt
+./gpte ask --questionfile test/samples/number-years-grow-30.txt --arith
 ```
 
-## Contributing
+These examples require GPT-4 token window size, code generation and reasoning:
+
+```bash
+./gpte ask --questionfile test/samples/msft-report-snippet.txt
+./gpte ask --questionfile test/samples/msft-report-snippet.txt --arith
 
-If you have suggestions for how changepilot-bot could be improved, or want to report a bug, open an issue! We'd love all and any contributions.
+./gpte ask --questionfile test/samples/msft-goog-report-snippets-compared.txt
+./gpte ask --questionfile test/samples/msft-goog-report-snippets-compared.txt --arith
 
-For more, check out the [Contributing Guide](CONTRIBUTING.md).
+./gpte ask --questionfile test/samples/gap-lulemon-financial-reports-compared.txt
+./gpte ask --questionfile test/samples/gap-lulemon-financial-reports-compared.txt --arith
+```
+
+With an appropriate model, entire problem sets can be evaluated using `./gpte eval`.
 
 ## License
 
-[ISC](LICENSE) © 2022 Matt Rothenberg
+This project is licensed under the terms of the MIT open source license. Please refer to [MIT](./LICENSE.txt) for the full terms.
+
+## Maintainers
+
+Maintainers: @dsyme, @wunderalbert, @johanrosenkilde
+
+## Support
+
+This project is a concluded technical investigation by GitHub Next. It is provided for reference.
+
+## Acknowledgement
+
+This project includes a modified version of the [`ASDiv.xml` dataset](https://github.com/chaochun/nlu-asdiv-dataset/blob/master/dataset/ASDiv.xml) for numeric calculation problems, adapted primarily to clarify the precisions and formats required in answers, see the [evaluation](docs/eval.md).

Dockerfile

@@ -0,0 +1,31 @@
+Thanks for helping make GitHub safe for everyone.
+
+# Security
+
+GitHub takes the security of our software products and services seriously, including all of the open source code repositories managed through our GitHub organizations, such as [GitHub](https://github.com/GitHub).
+
+Even though [open source repositories are outside of the scope of our bug bounty program](https://bounty.github.com/index.html#scope) and therefore not eligible for bounty rewards, we will ensure that your finding gets passed along to the appropriate maintainers for remediation.
+
+## Reporting Security Issues
+
+If you believe you have found a security vulnerability in any GitHub-owned repository, please report it to us through coordinated disclosure.
+
+**Please do not report security vulnerabilities through public GitHub issues, discussions, or pull requests.**
+
+Instead, please send an email to opensource-security[@]github.com.
+
+Please include as much of the information listed below as you can to help us better understand and resolve the issue:
+
+- The type of issue (e.g., buffer overflow, SQL injection, or cross-site scripting)
+- Full paths of source file(s) related to the manifestation of the issue
+- The location of the affected source code (tag/branch/commit or direct URL)
+- Any special configuration required to reproduce the issue
+- Step-by-step instructions to reproduce the issue
+- Proof-of-concept or exploit code (if possible)
+- Impact of the issue, including how an attacker might exploit the issue
+
+This information will help us triage your report more quickly.
+
+## Policy
+
+See [GitHub's Safe Harbor Policy](https://docs.github.com/en/github/site-policy/github-bug-bounty-program-legal-safe-harbor#1-safe-harbor-terms)

LICENSE

@@ -0,0 +1,13 @@
+# Support
+
+## How to file issues and get help
+
+This project uses GitHub issues to track bugs and feature requests. Please search the existing issues before filing new issues to avoid duplicates. For new issues, file your bug or feature request as a new issue.
+
+For help or questions about using this project, please file an issue.
+
+"gpt4-with-calc" is a reference implementation of the technique described in the report and is not actively developed but is maintained by GitHub staff. We will do our best to respond to support and community questions in a timely manner.
+
+## GitHub Support Policy
+
+Support for this project is limited to the resources listed above.