Skip to content

Commit 1429a4a

Browse files
huiyan2021huiyan
huiyan2021
and
huiyan
authored
add tf serving installation guide (#2303)
Co-authored-by: huiyan <[email protected]>
1 parent 47dd596 commit 1429a4a

File tree

1 file changed

+203
-0
lines changed

1 file changed

+203
-0
lines changed

‎docs/guide/tf_serving_install.md

Lines changed: 203 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,203 @@
1+
# Install TensorFlow Serving with Intel® Extension for TensorFlow*
2+
3+
[TensorFlow Serving](https://www.tensorflow.org/tfx/guide/serving) is an open-source system designed by Google that acts as a bridge between trained machine learning models and the applications that need to use them, streamlining the process of deploying and serving models in a production environment while maintaining efficiency and scalability.
4+
5+
## Install Model Server
6+
7+
### Install using Docker
8+
A good way to get started using TensorFlow Serving with Intel® Extension for TensorFlow* is with [Docker](https://www.docker.com/) containers.
9+
- Install Docker on Ubuntu 22.04
10+
```
11+
sudo apt install docker
12+
sudo apt install docker.io
13+
```
14+
- Pull docker image
15+
```
16+
# For CPU
17+
docker pull intel/intel-extension-for-tensorflow:serving-cpu
18+
19+
# For GPU
20+
docker pull intel/intel-extension-for-tensorflow:serving-gpu
21+
```
22+
23+
### Build from source
24+
> **Tips**:
25+
> - We recommend you put the source code of Intel® Extension for TensorFlow*, TensorFlow, and TensorFlow Serving in the same folder.
26+
> - Replace related paths with those on your machine.
27+
28+
#### 1. Build Intel® Extension for TensorFlow* C++ library
29+
Refer to [Intel® Extension for TensorFlow* for C++](https://intel.github.io/intel-extension-for-tensorflow/latest/docs/install/experimental/install_for_cpp.html) to build Intel® Extension for TensorFlow* C++ library
30+
> **Note:** When following this installation guide, you only need to build the Intel® Extension for TensorFlow* C++ library. You can ignore the other steps.
31+
32+
The generated `libitex_cpu_cc.so` or `libitex_gpu_cc.so` binary are found in the `intel_extension_for_tensorflow/bazel-bin/itex/` directory.
33+
34+
#### 2. Build TensorFlow Serving
35+
- Patch TensorFlow
36+
- Get TensorFlow with commit id specified by TensorFlow Serving: https://github.com/tensorflow/serving/blob/master/WORKSPACE#L28
37+
```
38+
# Exit intel-extension-for-tensorflow source code folder
39+
cd ..
40+
41+
# clone TensorFlow
42+
git clone https://github.com/tensorflow/tensorflow
43+
44+
# checkout specific commit id
45+
cd tensorflow
46+
git checkout xxxxx
47+
```
48+
- Add `alwayslink=1` for `kernels_experimental` library in local `tensorflow/tensorflow/c/BUILD` file:
49+
```
50+
tf_cuda_library(
51+
name = "kernels_experimental",
52+
srcs = ["kernels_experimental.cc"],
53+
hdrs = ["kernels_experimental.h"],
54+
copts = tf_copts(),
55+
visibility = ["//visibility:public"],
56+
deps = [
57+
...
58+
] + if_not_mobile([
59+
...
60+
]),
61+
alwayslink=1, # add this line
62+
)
63+
```
64+
- Patch TensorFlow Serving
65+
- Get TensorFlow Serving source code
66+
```
67+
# Exit tensorflow source code folder
68+
cd ..
69+
70+
git clone https://github.com/tensorflow/serving
71+
```
72+
- Patch TensorFlow Serving
73+
```
74+
cd serving
75+
patch -p1 -i ../intel-extension-for-tensorflow/third_party/tf_serving/serving_plugin.patch
76+
```
77+
- Update `serving/WORKSPACE` to use local TensorFlow
78+
Replace L24-L29 with below code to use local TensorFlow: https://github.com/tensorflow/serving/blob/master/WORKSPACE#L24
79+
```
80+
local_repository(
81+
name= "org_tensorflow",
82+
path = "path to local tensorflow source code",
83+
)
84+
```
85+
86+
- Build TensorFlow Serving
87+
```
88+
bazel build --copt="-Wno-error=stringop-truncation" --config=release //tensorflow_serving/model_servers:tensorflow_model_server
89+
```
90+
The generated `tensorflow_model_server` will be found in the `serving/bazel-bin/tensorflow_serving/model_servers/` directory.
91+
92+
### Build Docker image from Dockerfile
93+
Refer to [Intel® Extension for TensorFlow* Serving Docker Container Guide](../../docker/tensorflow-serving/README.md) to build docker image from dockerfile.
94+
95+
## Run sample
96+
- Train and export TensorFlow model
97+
```
98+
cd serving
99+
rm -rf /tmp/mnist
100+
python tensorflow_serving/example/mnist_saved_model.py /tmp/mnist
101+
```
102+
Now let's take a look at the export directory. You should find a directory named `1` that contains `saved_models.pb` file and `variables` folder.
103+
```
104+
ls /tmp/mnist
105+
1
106+
107+
ls /tmp/mnist/1
108+
saved_model.pb variables
109+
```
110+
- Load exported model with TensorFlow ModelServer plugged with Intel® Extension for TensorFlow*
111+
- Use Docker from Docker Hub
112+
```
113+
# For CPU
114+
docker run \
115+
-it \
116+
--rm \
117+
-p 8500:8500 \
118+
-e MODEL_NAME=mnist \
119+
-v /tmp/mnist:/models/mnist \
120+
intel/intel-extension-for-tensorflow:serving-cpu
121+
122+
# For GPU
123+
docker run \
124+
-it \
125+
--rm \
126+
-p 8500:8500 \
127+
-e MODEL_NAME=mnist \
128+
-v /tmp/mnist:/models/mnist \
129+
--device /dev/dri/ \
130+
-v /dev/dri/by-path/:/dev/dri/by-path/ \
131+
intel/intel-extension-for-tensorflow:serving-gpu
132+
```
133+
You will see:
134+
```
135+
plugin library "/itex/bazel-bin/itex/libitex_cpu_cc.so" load successfully!
136+
137+
plugin library "/itex/bazel-bin/itex/libitex_gpu_cc.so" load successfully!
138+
```
139+
140+
- Use tensorflow_model_server built from source
141+
```
142+
# cd tensorflow_model_server binary folder
143+
144+
# For CPU
145+
./tensorflow_model_server \
146+
--port=8500 \
147+
--rest_api_port=8501 \
148+
--model_name=mnist \
149+
--model_base_path=/tmp/mnist \
150+
--tensorflow_plugins=path_to_libitex_cpu_cc.so
151+
152+
# For GPU
153+
# source oneapi environment
154+
source oneapi_install_path/compiler/latest/env/vars.sh
155+
source oneapi_install_path/mkl/latest/env/vars.sh
156+
157+
./tensorflow_model_server \
158+
--port=8500 \
159+
--rest_api_port=8501 \
160+
--model_name=mnist \
161+
--model_base_path=/tmp/mnist \
162+
--tensorflow_plugins=path_to_libitex_gpu_cc.so
163+
```
164+
You will see:
165+
```
166+
plugin library "path_to_libitex_cpu_cc.so/libitex_cpu_cc.so" load successfully!
167+
168+
plugin library "path_to_libitex_gpu_cc.so/libitex_gpu_cc.so" load successfully!
169+
```
170+
171+
- Use Docker built from dockerfile
172+
```
173+
cd intel-extension-for-tensorflow source code folder
174+
175+
cd docker/tensorflow-serving
176+
177+
export MODEL_NAME=mnist
178+
export MODEL_DIR=/tmp/mnist
179+
180+
./run.sh [cpu/gpu]
181+
```
182+
You will see:
183+
```
184+
plugin library "/itex/itex-bazel-bin/bin/itex/libitex_cpu_cc.so" load successfully!
185+
186+
plugin library "/itex/itex-bazel-bin/bin/itex/libitex_gpu_cc.so" load successfully!
187+
```
188+
189+
- Test the server
190+
```
191+
pip install tensorflow-serving-api
192+
193+
cd serving
194+
python tensorflow_serving/example/mnist_client.py --num_tests=1000 --server=127.0.0.1:8500
195+
```
196+
You will see:
197+
```
198+
...
199+
200+
Inference error rate: xx.xx%
201+
```
202+
203+
Refer to [TensorFlow Serving Guides](https://www.tensorflow.org/tfx/serving/serving_basic) to learn more about how to use TensorFlow Serving.

0 commit comments

Comments
 (0)