The so-called waterfall method of building software was responsible for some very high-profile product flops in the 1980s and early 1990s. And today, enterprises run the same dramatic risks by using these outdated development methods on their AI initiatives.
Much like with traditional software development, as AI and ML initiatives take priority on the enterprise agenda, data scientists need to retire the outdated waterfall approach and replace it a more agile style of development. As application and system software developers learned in the 1990s, an agile approach enables a more flexible process, with significantly better chances of a successful deployment.
We think it’s time for Agile AI.
The Agile AI Way
Taking a page out of the software developer’s playbook, the Agile approach lets teams break AI initiatives into manageable bite-sized pieces. This technique allows teams to learn continuously and reprioritize where necessary, which ultimately leads to delivering value quicker and building the right model more efficiently.
How does Agile AI work?
Let’s say a publisher wants to offload moderation of user-generated photos to AI. Today, human moderators review contributed content to ensure nothing is posted that contains explicit material, copyrighted images, unauthorized advertising, etc. To train an algorithm to assume this role and know what to look for, it must be exposed to tens or hundreds of thousands of labeled or annotated photos.
If the publisher used the traditional waterfall approach, it would build the entire training data set, all hundreds of thousands of the labeled or annotated photos encompassing every category of acceptable and unacceptable content, before it started training the algorithm. Then it would expose the algorithm to the entire dataset and attempt to train it on everything all at once. If the algorithm struggled with anything, the entire initiative would be put on hold while the issue were discovered and all measures were taken to correct it.
On the other hand, organizations that follow an agile approach to machine learning will segment the problem space and tackle applying training data to the algorithm bit by bit, finding and correcting issues as they come up without slowing or stopping progress.
For example, the organization could focus on creating a training data set designed to teach the algorithm to detect just unauthorized advertising content. With the data prepared, the organization could train and then test the algorithm in an iterative fashion, until the model is exhibiting an acceptable confidence level. Check. That certified component of the model could then be put into production. The model would assume responsibility for detecting unauthorized advertising, and the human moderators who had been tracking this content, can move on to the next priority.
Software developers have been using the agile approach for decades, and data science teams that are adopting and deploying Agile AI are developing more accurate and valuable solutions faster.
For more information click here.