Rethinking AI Stack for Software Development

The Smart Developer's AI Stack

Over the last six months, my team has been using AI-assisted coding, which is now an essential part of our software development lifecycle (SDLC). We leverage enterprise LLMs to design applications and multi-tenant architectures, generate code, create database schemas, write SQL queries, develop unit tests, and prepare high-level design documents. Their value in accelerating software development has been undeniable.

As LLMs became more capable of generating complete solutions, another challenge emerged: token consumption. Since we relied on frontier models for almost every development activity, we frequently exhausted token and context limits, forcing developers to wait for token refreshes. This often led to delays in completing tasks within the planned timeline.

A considerable portion of the SDLC is spent on supporting activities such as:

• Documentation
• Unit test creation
• Test scenarios
• Bug fixing
• Code reviews
• Knowledge search

Using enterprise LLMs for these activities consumes a significant number of premium tokens and increases costs. Instead of relying on them for every task, organizations should adopt a more tactical approach.

Choosing the Right Model for the Right Task

My recommendation is straightforward: Use a local LLM for your development-related support activities and an enterprise LLM to build your application. Let an LLM manage complex code generation and create application architecture and design patterns. Once your main application is built, switch to a local LLM for support activities mentioned above.

A case in point. Once code is generated for a particular feature or module using an enterprise LLM, a local LLM can:

• Update the high and low-level design document
• Create a user guide for the developed module/feature
• Create unit tests
• Verify coding standards for code, SQL, and UI
• Create integrated test scenarios
• Suggest bug fixes

Local LLMs have evolved to a point where their quality and accuracy are good enough to manage the above points.

Another important advantage is privacy. Project documentation, test strategy, and application use guides, which reveal how the team works, remain within the organization.

Accessibility and Adoption

The essential part is driving adoption of the local LLM within the team. The easiest way is to host a good local LLM on a server with a GPU and create an API link.

The next step is to create a Visual Studio (VS) Code plugin that directly integrates with the local LLM API. Development teams use VS Code throughout the day for development activities. An API directly integrated into VS Code as a plugin is an effective way to drive adoption by tracking token usage of local LLMs.

With the VS Code plugin, developers can:

• Highlight the code and generate documentation
• Choose a developed logic or function to create unit tests
• Traverse to a defect tracker to identify possible fixes
• Use it as a Q&A to get answers
• Perform all these tasks without leaving the IDE.

This tactic improves productivity and gives better ROI on the enterprise LLM.

Enterprise models will continue to lead on the innovation front, while leveraging local LLMs for thousands of smaller support-related activities that consume developers’ time and premium AI tokens. Achieving the right balance by effectively combining both will lead to better productivity, cost optimization, and scalability.

Local LLMs are also useful as a lightweight API to:

• Run an internal knowledge search
• Look up code guidelines
• Check Vulnerability Assessment and Penetration Testing (VAPT) status
• Access code reusability repositories
• Pull project-specific YAML files for continuous deployment

Decision Guide for Local LLM Selection

If there is an opportunity to augment your infrastructure with GPUs, the following recommendations are worth considering:

In my experience development teams can reduce premium model usage by a significant extent by routing tasks as documentation, bug analysis, test scenarios to local LLMs, while enterprise or frontier LLMs can be reserved for complex development activities.

Spend your enterprise LLM tokens to build your main application rather than developing support activities.

Doing It Right

Most teams treat use frontier models for every task, which is the reason why token budgets run out before the sprint. The bigger LLM was never the blocker; how you allocate work between models was. Once support activities move to local LLMs, enterprise tokens move back to frontier-level reasoning, which includes architecture decisions, complex logic, and the code the business depends on. This helps local models close the quality gap and support better workforce decisions. The goal is not to replace enterprise models but to use local LLMs for the tasks they are best suited for.

Note to TechM: We have softened the language in this section to make the guidance more consultative and blog-appropriate. We have ensured the author's intent is preserved. Our copy:  

Selecting the right local LLM isn't about chasing the ‘best’ model. Instead, consider factors such as the available compute resources and the latency your environment can support. As a practical benchmark, a model that consistently delivers more than 15 tokens per second can provide a good developer experience.