I have created sample set of intents with phrases (five phrases per intent, and ten intents). Using this set of data to train and build classification model with TensorFlow and Oracle Chatbot machine learning. Once model is trained, classifying identical sample phrases with both TensorFlow and Oracle Chatbot to compare results. Using Oracle Chatbot with both Linguistic and Machine Learning models.
Summary:
1. Overall TensorFlow model performs better. The main reason for this - I was training TensorFlow model multiple times, until good learning output (minimized learning loss) was produced.
2. Oracle Chatbot doesn't return information about learning loss after training, this makes it hard to decide if training was efficient or no. As consequence - worse classification results, can be related to slightly less efficient training, simply because you don't get information about training efficiency
3. Classification results score: 93% TensorFlow, 87% Oracle Chatbot Linguistic model, 67% Oracle Chatbot Machine Learning. TensorFlow is better, but Oracle Chatbot Linguistic model is very close. Oracle Chatbot Machine Learning model can be improved, see point 2
Results table (click on it, to see maximized):
TensorFlow
List of intents for TensorFlow is provided in JSON file. Same intents are used to train model in Oracle Chatbot:
TensorFlow classification model is created by training 2-layer neural network. Once training is completed, it prints out total loss for the training. This allows to repeat training, until model is produced with optimal loss (as close as possible to 0): 0.00924 in this case:
TensorFlow classification result is good, it failed to classify only one sentence - "How you work?" This sentence is not directly related to any intent, although I should mention Oracle Chatbot Linguistic model is able to classify it. TensorFlow offers correct classification intent as second option, coming very close to correct answer:
Oracle Chatbot
Oracle Chatbot provides UI to enter intents and sample phrases - same set of intents with phrases is used as for TensorFlow:
Oracle Chatbot offers two training models - linguistic and machine learning based.
Once model is trained, there is no feedback about training loss. We can enter phrase and check intent classification result. Below is sample for Linguistic model classification failure - it fails to classify one of the intents, where sentence topic is not perfectly clear, however same intent is classified well by Oracle Chatbot Machine Learning model:
Oracle Chatbot Machine Learning model fails on another intent, where we want to check for hospital (hospital search) to monitor blood pressure. I'm sure if it would be possible to review training quality loss (may be in the next release?), we could decide to re-train model and get results close to TensorFlow. Classification with Oracle Chatbot Machine Learning model:
Summary:
1. Overall TensorFlow model performs better. The main reason for this - I was training TensorFlow model multiple times, until good learning output (minimized learning loss) was produced.
2. Oracle Chatbot doesn't return information about learning loss after training, this makes it hard to decide if training was efficient or no. As consequence - worse classification results, can be related to slightly less efficient training, simply because you don't get information about training efficiency
3. Classification results score: 93% TensorFlow, 87% Oracle Chatbot Linguistic model, 67% Oracle Chatbot Machine Learning. TensorFlow is better, but Oracle Chatbot Linguistic model is very close. Oracle Chatbot Machine Learning model can be improved, see point 2
Results table (click on it, to see maximized):
TensorFlow
List of intents for TensorFlow is provided in JSON file. Same intents are used to train model in Oracle Chatbot:
TensorFlow classification model is created by training 2-layer neural network. Once training is completed, it prints out total loss for the training. This allows to repeat training, until model is produced with optimal loss (as close as possible to 0): 0.00924 in this case:
TensorFlow classification result is good, it failed to classify only one sentence - "How you work?" This sentence is not directly related to any intent, although I should mention Oracle Chatbot Linguistic model is able to classify it. TensorFlow offers correct classification intent as second option, coming very close to correct answer:
Oracle Chatbot
Oracle Chatbot provides UI to enter intents and sample phrases - same set of intents with phrases is used as for TensorFlow:
Oracle Chatbot offers two training models - linguistic and machine learning based.
Once model is trained, there is no feedback about training loss. We can enter phrase and check intent classification result. Below is sample for Linguistic model classification failure - it fails to classify one of the intents, where sentence topic is not perfectly clear, however same intent is classified well by Oracle Chatbot Machine Learning model:
Oracle Chatbot Machine Learning model fails on another intent, where we want to check for hospital (hospital search) to monitor blood pressure. I'm sure if it would be possible to review training quality loss (may be in the next release?), we could decide to re-train model and get results close to TensorFlow. Classification with Oracle Chatbot Machine Learning model:
No comments:
Post a Comment