The New AI Stack: Integrating and Scaling AI Solutions with Modular Architecture

By Surabhi Sankhla

The rapid evolution of artificial intelligence (AI) technologies forces many companies to rethink their approach to integrating these capabilities. Most commercially deployed AI models were primarily predictive, but the rise of large language models (LLMs) has ushered in a new era of generative AI (GenAI). While LLMs represent the current state-of-the-art technology, the research and development of AI is ongoing. To remain competitive, it is vital for organizations to establish flexible and modular tech stacks that enable them to adopt the latest AI innovations quickly.

By decoupling the application layer from the AI and machine learning (ML) components, companies can swap out underlying models with minimal or no disruption to the customer experience. This architectural approach, along with open-source access to the latest models, democratizes AI. It empowers even non-technical organizations and small businesses to build sophisticated solutions that keep pace with the progress of these technologies.

The Commoditization of AI

Access to AI models is no longer limited to the largest tech firms with the resources and in-house expertise. The foundational AI technologies are now more accessible and cost-effective, giving even individuals direct access to AI through platforms like ChatGPT and Claude. LLMs are at the forefront of this trend, with smaller, more affordable versions like the GPT-4o Mini emerging alongside more powerful counterparts. Overall, the performance gap among various models is also narrowing, making the pricing more competitive than ever.

Related:AI Basics: A Quick Reference Guide for IT Professionals

Open access to AI models further accelerates this commoditization by lowering the entry barriers, allowing non-technology and smaller companies to bring AI-powered solutions into their organizations. Platforms like Hugging Face democratize access by releasing pre-trained open-source models that businesses adapt to their specific use cases. The development of industry-focused small language models (SLMs) and task-specific AI agents also allows smaller companies to integrate AI into their operations by significantly reducing the cost of training, deployment, and fine-tuning custom AI models.

The Monolith to Modular Architecture Spectrum

Apple's recent partnership with OpenAI demonstrates that even the most prominent tech giants will incorporate external AI services into their products. The Apple-OpenAI integration represents a more modular "plug and play" approach to AI rather than a monolithic in-house development model. Implementing modular or monolith architecture is not a binary choice, however, it's a spectrum. Apple's integration with OpenAI reflects higher modularity. In contrast, Amazon's approach with Alexa, with all components built and tightly integrated in-house, exemplifies a more monolithic architecture.

Related:AI Quiz 2024: Test Your AI Knowledge

While the monolith architecture offers a higher degree of control, it can hinder rapid innovation and adaptation. The risk that changes in one component will potentially affect the entire system limits the ability to easily leverage external AI capabilities. On the other hand, companies implementing a modular AI architecture can focus on what they do best — owning customer-facing aspects while integrating best-in-class AI capabilities from external providers. Modular design offers greater flexibility, enabling companies to leverage the latest advancements quickly and at scale. It also reduces the need for in-house development of all AI capabilities, further lowering entry barriers for smaller players with limited expertise and resources.

In a modular architecture, the tech layers communicate via application programming interfaces (APIs) and microservices that abstract the underlying implementation details. Stack layers can interact without needing to know the specifics of the other layer's internal operations. This decoupling isolates each layer and enables AI components to be swapped independently. The resulting plug-and-play approach lowers the costs and risks of making changes.

For instance, the retrieval augmented generation (RAG) layer can be updated independently to ensure access to the most current data sources. Similarly, the natural language understanding (NLU) layer can be easily replaced — switching from OpenAI to Gemini, for example — without changing components that rely on the NLU output. This flexibility enables the rapid adoption of cutting-edge AI technologies while maintaining system stability and performance.

Preparing to Go Modular

For many companies, especially those with extensive legacy systems, the upfront investment to transition to a modular architecture can be daunting, even considering the long-term benefits. Legacy data stored in outdated formats hinders a company's ability to adopt new AI technologies. While enterprises can get started with integrating common third-party applications, such as Salesforce and Slack, modernizing and migrating legacy data is critical for complete integration with new AI tools.

Similarly, another important early step is shifting to an API-centric and microservices-based architecture to reduce the number of tightly coupled systems. It is also crucial for companies to strengthen their data encryption and other safety measures, especially for sensitive customer data, to address the security issues of integrating with external AI services and ensure compliance with security regulations.

Lastly, investing in infrastructure, like cloud services, and considering advanced steps in the future, such as caching to reduce LLM calls, can further optimize performance.

The Modular AI Ecosystem

The modular AI ecosystem is a fluid environment comprising various players that contribute to the democratization and commoditization of AI technologies. Foundational model providers (e.g., ChatGPT and Koala) create core capabilities and specialized SLMs. Enterprise AI solution providers (e.g., Kore AI and Haptik) build prepackaged and customized domain and industry-specific solutions. AI service providers (e.g., HuggingFace and Scale AI) offer platforms to build AI models and provide services such as data labeling, prompt engineering, and fine-tuning AI models. Infrastructure players (e.g., AWS and Azure) provide cloud services to host AI models, data storage and management solutions, and high-performance computing resources.

This ecosystem facilitates the rapid innovation of AI technologies while broadening their reach. As the tech contributors build more advanced AI tools and services, modular architecture is the linchpin that enables seamless integration and accelerates growth. It makes it easier for innovators to push boundaries and for organizations of all sizes to adopt and scale AI solutions effectively.

Modular Architecture: Balancing Innovation and Stability

Adopting modular AI architectures offers significant opportunities but also presents challenges. While the transition and upfront investment can be costly and demanding, particularly for legacy-laden enterprises, the potential benefits — such as enhanced agility, lower costs, and easier access to specialized AI tools — are interesting.

For newer, more agile companies, the modular approach may be more organic and can simplify the adoption of cutting-edge AI technologies from the outset. As the AI ecosystem matures, it's critical for organizations to weigh the strategic value against the investments to ensure their technology stacks are truly adaptable and ready for the future.

About the Author:

Surabhi Sankhla is a vice president of product management and an expert in conversational AI. With more than 14 years of experience building digital platforms and solutions, she is currently leading the launch of an enterprise virtual assistant. During her tenure at Amazon, Surabhi built, launched, and expanded products for Amazon's Alexa. She also led several international digital transformation projects during her time at The Boston Consulting Group (BCG) and Uber. Surabhi has a bachelor's degree in electrical engineering from Indian Institute of Technology (IIT) Delhi and an MBA from Stanford. Connect with Surabhi on LinkedIn.