AI Assisted Decision Making of Security Review Needs for New Features

AI-assisted decision making can significantly improve accuracy of security review needs assessment for new software features by leveraging machine learning and natural language processing.

Key takeaways
  • AI can significantly improve the accuracy of security review needs assessment for new features in software development.
  • The absence of good quality data was a major issue, and the team experimented with different approaches, including Spark and natural language processing (NLP).
  • Stop words, such as “the”, “and”, “a”, are common words that don’t provide meaningful information and can be removed from the dataset.
  • Vectorization is a crucial step in converting text data into numbers that can be processed by machine learning algorithms.
  • The team used the technique of term frequency to vectorize their data, which worked well, but had a 2% error rate.
  • The F1 score, which measures the model’s accuracy, was used to evaluate the performance of the model.
  • The team also used an ensemble classifier, which combines the results of multiple weak models to produce a stronger model.
  • The model was trained on a dataset of 6,000 documents and was able to achieve a 98% accuracy rate.
  • The model was able to identify issues that required security review, but was also able to identify issues that did not require review.
  • The team noted that the model’s performance improved significantly when they used a larger dataset and a more complex model architecture.
  • The project demonstrated the potential of AI-powered decision making in software development, particularly in the area of security review needs assessment.