In-Depth
The Top 5 Mistakes Everyone Makes When Building AI Applications
Hardly a day seems to pass when we don't hear stories of someone's costly AI project failing to deliver the promised results. Interestingly, many of these failures follow a pattern. In fact, the problem is not usually the AI model itself, but rather the architectural and strategic decisions that have been made. As such, I wanted to take the opportunity to talk about five very common mistakes pertaining to AI application deployment.
Mistake 1: A Solution in Search of a Problem
Easily the number one mistake that I see being made with regard to AI application deployments is that of using AI just for the sake of using AI. We've all seen this. Simple workflows that don't even need AI are AI-enabled just because AI is trendy. This mistake often boils down to decision-makers saying, "we need to AI-enable our applications because that is what everybody else is doing," as opposed to trying to use AI to solve a particular business problem.
Mistake 2: Assuming That Bigger Is Better
The second mistake that everyone seems to make is that of assuming that bigger models are automatically better. Don't get me wrong -- there are undoubtedly situations where it makes sense to use a big model. Bigger models tend to do better than smaller models when it comes to complex reasoning and multi-step problem solving.
However, when it comes to performance, small models almost always outperform larger models. It also usually costs less to host a small model than a large one. This holds true whether you are hosting the model in the cloud or in your own data center. Larger models consume far more system resources and incur more latency than smaller models.
As a best practice, organizations should select a model based on how well it fits their needs, not based on the model size. I usually recommend that an organization start with the smallest model that seems as though it might be a good fit and then upgrade to a larger model only if necessary.
Mistake 3: Ignoring Data Quality
Another mistake that is often made is focusing on quantity instead of quality when it comes to the training data that you use. It is absolutely true that AI models need to be trained against large datasets. However, the quality of the training data matters -- a lot.
Early in my IT career, someone told me about the GIGO principle. GIGO stands for Garbage In, Garbage Out. The idea was that no matter how good your application might be, the output that it delivers will only be as good as the data that has been entered into the application. Even though I first heard about the GIGO principle more than 30 years ago, I honestly believe that this principle is even more relevant today than it was back then.
Simply put, AI systems are only as good as the data that they are trained on. If your data is outdated, inconsistent, contradictory, or noisy, then any AI that is trained on that dataset will mirror those same problems.
Mistake 4: Failing to Rigorously Test the Application
On the surface, the idea of inadequate testing probably seems absurd. After all, stringent testing has always been a part of the software development process. Even so, I have seen several real-world examples of AI-enabled applications that were rushed into production following what amounted to basic bug fixes and giving the software a quick once-over.
In some ways, the idea of rushing an AI application into production with minimal testing isn't all that shocking. Organizations have been investing heavily in AI applications, and so there is pressure to try to achieve the quickest possible ROI. There is also pressure to get applications into production quickly in order to try to gain a competitive advantage. Besides, modern software development processes often focus on agility and rapid deployments with the idea that problems can always be patched later on.
When it comes to AI-enabled applications, however, the problem is that failures are not always immediately obvious. Business decisions might be made based on AI hallucinations, with the organization only realizing the problem after the damage has already been done.
Mistake 5: Underestimating Ongoing Maintenance
Some conventional applications fall into the category of "set it and forget it." As an example, there is a hardware store near where I live that still runs its inventory software on an old Windows 2000 machine. That system has been in place for decades, but works perfectly. The store has adopted the "if it's not broken, don't fix it" philosophy.
Conversely, AI applications definitely do not fit into this category. AI applications are like living systems that require continuous care. Without constant maintenance, the workload can begin to quietly degrade shortly after it is brought into production. This degradation can occur as a result of model drift, changing data (business conditions change and the current data no longer resembles the data that the model was trained on), or even scalability problems. Additionally, costs increase as usage scales and as maintenance begins to be required. Organizations would do well to understand that the cost models that apply to traditional software development and maintenance do not align well with AI workloads.
About the Author
Brien Posey is a 22-time Microsoft MVP with decades of IT experience. As a freelance writer, Posey has written thousands of articles and contributed to several dozen books on a wide variety of IT topics. Prior to going freelance, Posey was a CIO for a national chain of hospitals and health care facilities. He has also served as a network administrator for some of the country's largest insurance companies and for the Department of Defense at Fort Knox. In addition to his continued work in IT, Posey has spent the last several years actively training as a commercial scientist-astronaut candidate in preparation to fly on a mission to study polar mesospheric clouds from space. You can follow his spaceflight training on his Web site.