v1.0 cherry picks (#450)

* Introduced readies submodule (#377)

* Introduced readies submodule

* Fix in paella

* Updated get_deps.sh and docs

* [WIP] add C API reference to docs using auto tool  (#368)

* [add] add C API reference to docs using auto tool ( python3 generate_llapi_reference.py )

* [add] generating C API reference on the fly

* [add] moving api_reference under Developer Nav

Co-authored-by: Luca Antiga <[email protected]>

* [WIP] Enable AI.SCRIPTRUN on AI.DAGRUN* (#383)

* [add] decoupled scriptrun command parsing from runtime scriptrun.

* [add] split positive/negative tests on pytorch scriptrun.

* [add] refactor AI.DAGRUN_RO and AI.DAGRUN to use the same code base (with read/write modes)

* [add] added positive and negative tests for dagrun with scriptrun

* [add] updated documentation to reflect scriptrun support on dagrun

* [add] added example enqueuing multiple SCRIPTRUN and MODELRUN commands within a DAG

* Add support for variadic arguments to SCRIPT (#395)

* Add support for variadic arguments to SCRIPT

* Add negative errors

* atomic ref count (#403)

* Multi-platform build (#398)

* CircleCI: increased GPU test timeout to 40m (#404)

* CI: platform-build-defs -> on-master-and-version-tags (#405)

* Llapi updates (#400)

* added low level api return redis types

* added variadic to llapi

* fixed memory issue for params array re-alloc

* Avoid splitting outputs in batches when nbatches == 1 (#406)

* Avoid splitting outputs in batches when nbatches == 1

* Add batch size checks

* Fix batch checks

* Update readies

* Add bad batching test

* Update README.md

* CircleCI: fixed redis installation in coverage builds (#423)

* Docker6.0.5 (#427)

* Update Dockerfile

* Update Dockerfile.arm

* Update Dockerfile.gpu

* submodule pull moved up (#428)

* Update mkdocs.yml (#432)

* [add] Add relevant RedisAI config entries to the INFO output, so that standard monitoring systems would be able to monitor them. (#396)

* Updated support email and Orobix to Tensorwork on ramp file (#436)

* [add] updated support email and Orobix to Tensorwork on ramp file

Co-authored-by: Sherin Thomas <[email protected]>

* CircleCI: Fixed problem with GPU testing (#440)

* Fixed platforms build problem (#441)

* Fixed platforms build problem

* Build: updated Redis versions

* Build: another Redis version update

* Dependencies in ramp.yml (#444)

* Fixed flagged as "getkeys-api" during the registration ( AI.DAGRUN, AI.DAGRUN_RO, AI.MODELRUN, AI.SCRIPTRUN ) (#438)

* [wip] wip on fixing the commands being flagged as getkeys-api during the registration. ( AI.MODELRUN, AI.SCRIPTRUN, AI.DAGRUN, AI.DAGRUN_RO )

* [add] pytorch oss cluster tests passing

* [add] TensorFlow lite tests passing on oss cluster

* [add] ONNX tests passing on oss cluster

* [add] TensorFlow tests passing on oss cluster

* [wip] wip on dag

* [add] DAG tests passing on oss cluster

* [add] enabling oss cluster tests on CI

* [add] bumping rmbuilder version from 6.0.1 to 6.0.5 on circleci

* [fix] fixed potential invalid mem accesses on RedisAI_DagRunSyntaxParser, RAI_parseDAGPersistArgs

* [fix] fixed RAI_FreeDagOp wrongfully calling RedisModule_Free on dagOp->inkeys, dagOp->outkeys

* [fix] fixed dictKey allocation on RAI_parseDAGLoadArgs

* [add] alter Sanitizer tests to accommodate keys on the same slot.

* [add] extracted the methods to respond to RedisModule_IsKeysPositionRequest() from main code of dagrun,modelrun, and scriptrun to specific methods

* [fix] Fixes per PR review

* [fix] fixes per PR review

* [fix] fix per CI ascii conversion error on test_dagro_modelrun_financialNet_no_writes_multiple_modelruns

* Safely add to arrays + fix for #443 (#449)

* Make sure we reassign the pointer in array_append

* Fix case-sensitive comparison for devicestr

* Fix sanitizer tests

(cherry picked from commit 5c7813e)

* fixed tests messages

* Shallow copy persisted tensor

* Resolve log format warnings

* Fix error message and test

* Fix memory management in local context dict

* Fixed artifacts handling + added tests logs aggregation (#445)

* Snapshot packages are placed into http://redismodules.s3.amazonaws.com/redisai/snapshots. Private branches (i.e., non-master) are also supported, just need to enable deploy-snapshot in config.yml to fire on on-any-branch rather than just on-master.
* Release packages are placed into http://redismodules.s3.amazonaws.com/redisai, and it's going to be crowded in there, so we may want to consider putting each version into its own directory.
* `BB` in-source breakpoints are now supported (in `DEBUG=1` builds): put `BB;` inside the code and it will stop if you're running under gdb.
* Tests results in CircleCI are now aggregated as artifacts.

* Build: setuptools-related fix (#455)

(cherry picked from commit d1fcd0d)

Co-authored-by: Rafi Einstein <[email protected]>
Co-authored-by: Luca Antiga <[email protected]>
Co-authored-by: DvirDukhan <[email protected]>
Co-authored-by: Guy Korland <[email protected]>
Co-authored-by: Sherin Thomas <[email protected]>

Author: 6 people

.circleci/config.yml

@@ -30,7 +30,7 @@ commands:
           command: |
             ./opt/readies/bin/getpy3
             BREW_NO_UPDATE=1 ./opt/system-setup.py
-            # ./opt/readies/bin/getredis -v 6 --force
+            ./opt/readies/bin/getredis -v 6 --force
             ./get_deps.sh cpu
       - save_cache:
           paths:
@@ -46,12 +46,17 @@ commands:
           no_output_timeout: 20m
       - run:
           name: Package
-          command: make -C opt pack SHOW=1
+          command: |
+            make -C opt pack SHOW=1
+            (cd bin/artifacts; tar -cf snapshots-<<parameters.platform>>.tar snapshots/)
       - persist_to_workspace:
           root: bin/
           paths:
             - artifacts/*.zip
             - artifacts/*.tgz
+            - artifacts/*.tar
+            # - artifacts/shapshots/*.zip
+            # - artifacts/shapshots/*.tgz
       - store_artifacts:
           path: test/logs
 
@@ -71,16 +76,27 @@ commands:
           name: Build for platform
           command: |
             docker login -u redisfab -p $DOCKER_REDISFAB_PWD
-            cd opt/build/docker
+            pushd opt/build/docker
             #@@ make build publish $(./version-params) CPU=1 OSNICK=<<parameters.platform>> X64=1 ARTIFACTS=1 TEST=1 VERBOSE=1
             make build publish $(./version-params) CPU=1 OSNICK=<<parameters.platform>> X64=1 ARTIFACTS=1 VERBOSE=1
             make build publish $(./version-params) GPU=1 OSNICK=<<parameters.platform>> X64=1 ARTIFACTS=1 VERBOSE=1
+            popd > /dev/null
+            logstar=bin/artifacts/tests-logs-cpu.tgz
+            logsdir=tests/logs/cpu
+            mkdir -p $logsdir
+            if [[ -e $logstar ]]; then tar -C $logsdir -xzf $logstar; fi
+            (cd bin/artifacts; tar -cf snapshots-<<parameters.platform>>.tar snapshots/)
           no_output_timeout: 40m
       - persist_to_workspace:
           root: bin/
           paths:
             - artifacts/*.zip
             - artifacts/*.tgz
+            - artifacts/*.tar
+            # - artifacts/shapshots/*.zip
+            # - artifacts/shapshots/*.tgz
+      - store_artifacts:
+          path: test/logs
 
   deploy-steps:
     parameters:
@@ -95,7 +111,7 @@ commands:
 jobs:
   build-debian:
     docker:
-      - image: redisfab/rmbuilder:6.0.1-x64-buster
+      - image: redisfab/rmbuilder:6.0.5-x64-buster
     steps:
       - build-steps:
           platform: debian
@@ -112,7 +128,7 @@ jobs:
 
   coverage:
     docker:
-      - image: redisfab/rmbuilder:6.0.1-x64-buster
+      - image: redisfab/rmbuilder:6.0.5-x64-buster
     steps:
       - checkout
       - run:
@@ -136,7 +152,7 @@ jobs:
       - run:
           name: Test with coverage
           command: |
-            make -C opt test SHOW=1 COV=1
+            make -C opt test SHOW=1 COV=1 CLUSTER=1
             make -C opt cov-upload
           no_output_timeout: 20m
 
@@ -202,22 +218,56 @@ jobs:
             docker run --gpus all -v $HOME/tests:/build/test/logs -it --rm redisai-gpu:latest-x64-bionic-test
           no_output_timeout: 40m
       - store_artifacts:
-          path: tests
+          path: test/log
 
   deploy-artifacts:
     parameters:
-      package:
+      location:
         type: string
     docker:
-      - image: redisfab/rmbuilder:6.0.1-x64-buster
+      - image: redisfab/rmbuilder:6.0.5-x64-buster
     steps:
       - attach_workspace:
           at: workspace
       - run:
           name: Deploy to S3
           command: |
-            cd workspace
-            aws s3 cp artifacts/ s3://redismodules/$PACKAGE_NAME/ --acl public-read --recursive --exclude "*" --include "*.zip" --include "*.tgz"
+            cd "workspace/artifacts/<<parameters.location>>"
+            for f in *.zip snapshot/*.tgz; do
+              aws s3 cp $f s3://redismodules/$PACKAGE_NAME/<<parameters.location>>/ --acl public-read
+            done
+
+  deploy-snapshot:
+    docker:
+      - image: redisfab/rmbuilder:6.0.5-x64-buster
+    steps:
+      - attach_workspace:
+          at: workspace
+      - run:
+          name: Deploy Snapshots to S3
+          command: |
+            cd workspace/artifacts
+            for f in snapshots-*.tar; do
+                tar xf $f
+            done
+            cd snapshots
+            for f in *.zip *.tgz; do
+              aws s3 cp --no-progress $f s3://redismodules/$PACKAGE_NAME/snapshots/ --acl public-read
+            done
+
+  deploy-release:
+    docker:
+      - image: redisfab/rmbuilder:6.0.5-x64-buster
+    steps:
+      - attach_workspace:
+          at: workspace
+      - run:
+          name: Deploy Releases to S3
+          command: |
+            cd workspace/artifacts
+            for f in *.zip *.tgz; do
+              aws s3 cp --no-progress $f s3://redismodules/$PACKAGE_NAME/ --acl public-read
+            done
 
 
 on-any-branch: &on-any-branch
@@ -296,14 +346,10 @@ workflows:
       - build-multiarch-docker:
           <<: *never # temporarily disabled
           # <<: *on-version-tags
-      - deploy-artifacts:
-          name: deploy-branch
-          package: branch
+      - deploy-snapshot:
           <<: *after-platform-builds
-          <<: *on-any-branch
-      - deploy-artifacts:
-          name: deploy-release
-          package: release
+          <<: *on-master
+      - deploy-release:
           <<: *after-platform-builds
           <<: *on-version-tags
 

.github/workflows/deploy-docs.yaml

@@ -21,6 +21,8 @@ jobs:
         python-version: '3.x'
     - name: Display Python version
       run: python -c "import sys; print(sys.version)"
+    - name: Install Doxygen
+      run: apt-get install -y doxygen
     - name: Install docs dependencies
       run: pip install -r docs/requirements.txt
     - name: Get entire project

CMakeLists.txt

@@ -39,8 +39,10 @@ set(CMAKE_C_FLAGS_RELEASE "${CMAKE_C_FLAGS_RELEASE} -O3")
 set(CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS_RELEASE} -O3")
 
 # Add -fno-omit-frame-pointer to avoid seeing incomplete stack traces
-set(CMAKE_C_FLAGS_DEBUG "${CMAKE_C_FLAGS_DEBUG} -g -ggdb -fno-omit-frame-pointer -DVALGRIND")
-set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} -g -ggdb -fno-omit-frame-pointer -DVALGRIND")
+set(CMAKE_COMMON_FLAGS_DEBUG "-g -ggdb -fno-omit-frame-pointer -D_DEBUG -DVALGRIND -include \
+    ${CMAKE_CURRENT_SOURCE_DIR}/src/common.h -I${CMAKE_CURRENT_SOURCE_DIR}/opt")
+set(CMAKE_C_FLAGS_DEBUG "${CMAKE_C_FLAGS_DEBUG} ${CMAKE_COMMON_FLAGS_DEBUG}")
+set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} ${CMAKE_COMMON_FLAGS_DEBUG}")
 
 #----------------------------------------------------------------------------------------------
 

Dockerfile

@@ -1,6 +1,6 @@
 # BUILD redisfab/redisai:${VERSION}-cpu-${ARCH}-${OSNICK}
 
-ARG REDIS_VER=6.0.1
+ARG REDIS_VER=6.0.5
 
 # OSNICK=bionic|stretch|buster
 ARG OSNICK=buster
@@ -31,7 +31,7 @@ COPY --from=redis /usr/local/ /usr/local/
 COPY ./opt/ opt/
 COPY ./test/test_requirements.txt test/
 
-RUN ./opt/readies/bin/getpy3
+RUN PIP=1 FORCE=1 ./opt/readies/bin/getpy3
 RUN ./opt/system-setup.py
 
 ARG DEPS_ARGS=""
@@ -40,17 +40,24 @@ RUN if [ "$DEPS_ARGS" = "" ]; then ./get_deps.sh cpu; else env $DEPS_ARGS ./get_
 
 ARG BUILD_ARGS=""
 ADD ./ /build
-RUN set -e ;\
+RUN bash -c "set -e ;\
     . ./opt/readies/bin/sourced ./profile.d ;\
-    make -C opt build $BUILD_ARGS SHOW=1
+    make -C opt build $BUILD_ARGS SHOW=1"
 
 ARG PACK
 ARG TEST
 
 RUN mkdir -p bin/artifacts
-RUN if [ "$PACK" = "1" ]; then make -C opt pack; fi
+RUN set -e ;\
+    if [ "$PACK" = "1" ]; then make -C opt pack; fi
 
-RUN if [ "$TEST" = "1" ]; then TEST= make -C opt test $BUILD_ARGS NO_LFS=1; fi
+RUN set -e ;\
+    if [ "$TEST" = "1" ]; then \
+        TEST= make -C opt test $BUILD_ARGS NO_LFS=1 ;\
+        if [[ -d test/logs ]]; then \
+            tar -C test/logs -czf bin/artifacts/test-logs-cpu.tgz . ;\
+        fi ;\
+    fi
 
 #----------------------------------------------------------------------------------------------
 FROM redisfab/redis:${REDIS_VER}-${ARCH}-${OSNICK}

Dockerfile.arm

@@ -1,6 +1,6 @@
 # BUILD redisfab/redisai:${VERSION}-cpu-${ARCH}-${OSNICK}
 
-ARG REDIS_VER=6.0.1
+ARG REDIS_VER=6.0.5
 
 # OSNICK=bionic|stretch|buster
 ARG OSNICK=buster

Dockerfile.gpu

@@ -1,16 +1,16 @@
 # BUILD redisfab/redisai:${VERSION}-gpu-${ARCH}-${OSNICK}
 
-ARG REDIS_VER=6.0.1
+ARG REDIS_VER=6.0.5
 
 # OSNICK=bionic|centos7|centos6
 ARG OSNICK=bionic
 
-# ARCH=x64|arm64v8|arm32v7
-ARG ARCH=x64
-
 # OS=ubuntu18.04|ubuntu16.04|centos7
 ARG OS=ubuntu18.04
 
+# ARCH=x64|arm64v8|arm32v7
+ARG ARCH=x64
+
 ARG CUDA_VER=10.1-cudnn7
 
 ARG PACK=0
@@ -22,10 +22,11 @@ FROM nvidia/cuda:${CUDA_VER}-devel-${OS} AS builder
 
 ARG OSNICK
 ARG OS
+ARG ARCH
 ARG REDIS_VER
 ARG CUDA_VER
 
-RUN echo "Building for ${OSNICK} (${OS}) [with Redis ${REDIS_VER}]"
+RUN echo "Building for ${OSNICK} (${OS}) for ${ARCH} [with Redis ${REDIS_VER}]"
 
 ENV NVIDIA_VISIBLE_DEVICES all
 ENV NVIDIA_DRIVER_CAPABILITIES compute,utility
@@ -36,7 +37,7 @@ COPY --from=redis /usr/local/ /usr/local/
 COPY ./opt/ opt/
 COPY ./test/test_requirements.txt test/
 
-RUN ./opt/readies/bin/getpy3
+RUN PIP=1 FORCE=1 ./opt/readies/bin/getpy3
 RUN ./opt/system-setup.py
 
 ARG DEPS_ARGS=""
@@ -45,17 +46,24 @@ RUN if [ "$DEPS_ARGS" = "" ]; then ./get_deps.sh gpu; else env $DEPS_ARGS ./get_
 
 ARG BUILD_ARGS=""
 ADD ./ /build
-RUN set -e ;\
+RUN bash -c "set -e ;\
     . ./opt/readies/bin/sourced ./profile.d ;\
-    make -C opt build GPU=1 $BUILD_ARGS SHOW=1
+    make -C opt build GPU=1 $BUILD_ARGS SHOW=1"
 
 ARG PACK
 ARG TEST
 
 RUN mkdir -p bin/artifacts
-RUN if [ "$PACK" = "1" ]; then make -C opt pack GPU=1; fi
+RUN set -e ;\
+    if [ "$PACK" = "1" ]; then make -C opt pack GPU=1; fi
 
-RUN if [ "$TEST" = "1" ]; then TEST= make -C opt test GPU=1 $BUILD_ARGS NO_LFS=1; fi
+RUN set -e ;\
+    if [ "$TEST" = "1" ]; then \
+        TEST= make -C opt test GPU=1 $BUILD_ARGS NO_LFS=1 ;\
+        if [[ -d test/logs ]]; then \
+            tar -C test/logs -czf bin/artifacts/test-logs-gpu.tgz . ;\
+        fi ;\
+    fi
 
 #----------------------------------------------------------------------------------------------
 FROM nvidia/cuda:${CUDA_VER}-runtime-${OS}

Dockerfile.gpu-test

@@ -1,6 +1,6 @@
 # BUILD redisfab/redisai:${VERSION}-gpu-${ARCH}-${OSNICK}-test
 
-ARG REDIS_VER=6.0.1
+ARG REDIS_VER=6.0.5
 
 # OSNICK=bionic|centos7|centos6
 ARG OSNICK=bionic
@@ -29,11 +29,12 @@ COPY --from=redis /usr/local/ /usr/local/
 COPY ./opt/ opt/
 COPY ./test/test_requirements.txt test/
 
-RUN ./opt/readies/bin/getpy3
-RUN ./opt/system-setup.py
+RUN PIP=1 VENV=1 FORCE=1 ./opt/readies/bin/getpy3
+
 RUN set -e ;\
-	python3 -m virtualenv venv --system-site-packages;\
+	python3 -m virtualenv venv --system-site-packages ;\
 	. venv/bin/activate ;\
+	python -m pip install pip==19.3.1 ;\
 	./opt/system-setup.py
 
 ARG DEPS_ARGS=""

README.md

@@ -130,6 +130,7 @@ RedisAI currently supports PyTorch (libtorch), Tensorflow (libtensorflow), Tenso
 | 0.3.1   | 1.1.0   | 1.12.0     | None   | 0.4.0         |
 | 0.4.0   | 1.2.0   | 1.14.0     | None   | 0.5.0         |
 | 0.9.0   | 1.3.1   | 1.14.0     | 2.0.0  | 1.0.0         |
+| 1.0.0   | 1.5.0   | 1.15.0     | 2.0.0  | 1.2.0         |
 | master  | 1.5.0   | 1.15.0     | 2.0.0  | 1.2.0         |
 
 Note: Keras and TensorFlow 2.x are supported through graph freezing. See [this script](https://github.com/RedisAI/RedisAI/blob/master/test/test_data/tf2-minimal.py) to see how to export a frozen graph from Keras and TensorFlow 2.x. Note that a frozen graph will be executed using the TensorFlow 1.15 backend. Should any 2.0 ops be not supported on the 1.15 after freezing, please open an Issue.