Tensorflow 1 vs Tensorflow 2 C-API
With the new tensorflow version 2, there was a lot of changes in the python API to train and test the models. Similarly, we were expecting major changes in tensorflow C-API. But it seems that there are not as much as we had expected. Most of the changes are the side effects of how we train the model and how they are saved. In the following post I try to document the changes compared to tensorflow version 1 C-API, and inference an object detection model (SSD) trained with the google API.
Tensorlfow 2.4 C-API was used in this post.
Tensorflow model formats
We can save tensorflow model in many different formats. To be able to load the C-API, we could use the frozen .pb files in the tensorflow version 1 (tfv1). But with tensorflow version 2 (tfv2), we require the model to be saved in the SavedModel format. For the single shot detector model we used the exporter_main_v2.py script from here.
Loading the model
Loading the model in tfv1 included opening the model file, importing the graph from the buffer file and finally creating the session. But in tfv2 it is simple and a single line to create the session and load the graph. The codes for both tfv1 and tfv2 are given below.
Tensorflow V1
void Network::LoadGraph(std::string modelPath){
TF_Buffer* buffer = ReadBufferFromFile(modelPath);
if (buffer == nullptr) {
throw std::invalid_argument(“ERROR model path not found %s !”);
}
TF_Status* status = TF_NewStatus();
TF_ImportGraphDefOptions* opts = TF_NewImportGraphDefOptions();
graph = TF_NewGraph();
TF_GraphImportGraphDef(graph, buffer, opts, status);
TF_DeleteImportGraphDefOptions(opts);
TF_DeleteBuffer(buffer);
if (TF_GetCode(status) != TF_OK) {
TF_DeleteGraph(graph);
graph = nullptr;
}
TF_DeleteStatus(status);
// create session from graph
status = TF_NewStatus();
TF_SessionOptions* options = TF_NewSessionOptions();
session = TF_NewSession(graph, options, status);
TF_DeleteSessionOptions(options);
}Tensorflow V2
void Network::LoadGraph(std::string modelPath)
{
graph = TF_NewGraph();
TF_Status* Status = TF_NewStatus();
TF_SessionOptions* SessionOpts = TF_NewSessionOptions();
TF_Buffer* RunOpts = NULL;
const char* tags = "serve";
int ntags = 1;
char* path = const_cast<char*>(modelPath.c_str()); session = TF_LoadSessionFromSavedModel(SessionOpts, RunOpts, path, &tags, ntags, graph, NULL, Status);
if (TF_GetCode(Status) == TF_OK)
{
printf("Tensorflow 2x Model loaded OK\n");
}
else
{
printf("%s", TF_Message(Status));
}
TF_DeleteSessionOptions(SessionOpts);
TF_DeleteStatus(Status);
}
Input/output tensors
In tfv1 we had the graph easily available in python API. So, printing all the operation and tensors in the graph was possible using the code below.
for op in tf.get_default_graph().get_operations():
print(str(op.name))To run the session using the python API for tf v1, we required the tensor names. So, it was similar to what we required for the C-API. The tensor name for single shot detector trained using the google API were:
- input tensor → image_tensor:0
- output tensor (bounding boxes)→ detection_boxes:0
- output tensor (scores) → detection_scores:0
- output tensor (classes) → detection_classes:0
There must also be a way we can access the graph in tfv2 python API and get the operation tensors. But we were able to get operations and tensors using a tool that tensorflow provides when we install tensorflow called the ‘saved_model_cli’ (detail information can be found here).
Here we show the way to find the input/output tensor name using the saved_model_cli for an SSD trained with google API. The input and the output tensors were as follows:
- input tensor → serving_default_input_tensor:0
- output tensor (bounding boxes) → StatefulPartitionedCall:0
- output tensor (scores) → StatefulPartitionedCall:3
- output tensor (classes) → StatefulPartitionedCall:2
>> saved_model_cli show --dir <<path of your model>>
The given SavedModel contains the following tag-sets:
'serve'>> saved_model_cli show --dir <<path of your model>> --tag serve
The given SavedModel MetaGraphDef contains SignatureDefs with the following keys:
SignatureDef key: "__saved_model_init_op"
SignatureDef key: "serving_default">> saved_model_cli show --dir <<path of your model>> --tag serve --signature serving_defaultThe given SavedModel SignatureDef contains the following input(s):
inputs['input_tensor'] tensor_info:
dtype: DT_UINT8
shape: (1, -1, -1, 3)
name: serving_default_input_tensor:0
The given SavedModel SignatureDef contains the following output(s):
outputs['detection_boxes'] tensor_info:
dtype: DT_FLOAT
shape: (1, 100, 4)
name: StatefulPartitionedCall:0
outputs['detection_boxes_strided'] tensor_info:
dtype: DT_FLOAT
shape: (1, 100, 4)
name: StatefulPartitionedCall:1
outputs['detection_classes'] tensor_info:
dtype: DT_FLOAT
shape: (1, 100)
name: StatefulPartitionedCall:2
outputs['detection_scores'] tensor_info:
dtype: DT_FLOAT
shape: (1, 100)
name: StatefulPartitionedCall:3
outputs['num_detections'] tensor_info:
dtype: DT_FLOAT
shape: (1)
name: StatefulPartitionedCall:4
Method name is: tensorflow/serving/predict
We can also print all the operations of the graph using the C-API.
while ((oper = TF_GraphNextOperation(graph, &pos)) != nullptr) {
printf(TF_OperationName(oper));
printf("\n");
}With the available tensor name, we can get the required input/output tensor from the graph as show in code below. Be careful with the index of the tensor because the name of the tensor for bounding box, confidence and classes are the same and it only differs with the index. StatefulPartitionedCall with 0 index will give us the bounding box tensor while StatefulPartitionedCall with index 3 will give us scores tensor.
#For Bounding box
TF_Output bbox= { TF_GraphOperationByName(graph, "StatefulPartitionedCall"),0};
if (bbox.oper == NULL)
printf("ERROR: Failed TF_GraphOperationByName StatefulPartitionedCall\n");
else
printf("TF_GraphOperationByName StatefulPartitionedCall is OK\n");#For scores
TF_Output scores= { TF_GraphOperationByName(graph, "StatefulPartitionedCall"),3};
if (bbox.oper == NULL)
printf("ERROR: Failed TF_GraphOperationByName StatefulPartitionedCall_3\n");
else
printf("TF_GraphOperationByName StatefulPartitionedCall_3 is OK\n");
Run the session
Now that we have the session, graph and the required input output tensors, we can simply run the exact code we used in tfv1 to run the session.
TF_SessionRun(session, nullptr,
&input_tensors[0], &input_values[0], input_values.size(),
&output_tensors[0], &output_values[0], 3, //3 is the number of outputs count..
//Output, &output_values[0], 1,
nullptr, 0, nullptr, status
);Free Allocated memory
We need to free the memory allocated for the graph, session and the status. It is also necessary to delete the allocated space for the output.
TF_DeleteGraph(Graph);
TF_DeleteSession(Session, Status);
TF_DeleteStatus(Status);The code to delete the allocated space for the output is given below.
for (auto& t : output_values) {
TF_DeleteTensor(t);
}
output_values.clear();
output_values.shrink_to_fit();Complete Visual Studio Solution
https://github.com/danishshres/ssd_c_api/tree/tf_2