RAG, fine-tuning, and prompt engineering are everywhere — but what does 'production-ready AI' look like for a 10–50 person startup? We spoke to 15 hiring managers.
Since ChatGPT launched in late 2022, every job description from a startup has added 'AI experience preferred' somewhere. But when we asked 15 hiring managers at Indian startups — mostly Series A and B — what that actually means, the answers were surprisingly specific, and quite different from what most candidates are preparing.
It doesn't mean using ChatGPT. Every candidate uses ChatGPT. What startups mean is: can you build a system that uses an LLM reliably in a production environment, where real users depend on it and errors have consequences?
The gap between 'I've used AI tools' and 'I've built AI-powered features' is enormous in practice. The former is universal; the latter is still genuinely rare among fresh and junior engineers. That gap is where the opportunity is.
Production-ready AI means understanding cost (token pricing compounds fast), latency (users notice anything over 2 seconds), and failure modes (hallucinations, context limits, rate limits). Candidates who have thought about these things stand out immediately.
11 of the 15 hiring managers specifically mentioned Retrieval-Augmented Generation. Several said it was more important than knowing how to fine-tune a model. The reason: most startup AI products are built on RAG, not fine-tuning.
What they want to see: the ability to build a retrieval pipeline from scratch — document ingestion, chunking strategy, embedding generation, and vector storage. Understanding the trade-offs between different chunking approaches and retrieval methods (dense, sparse, hybrid) is a real differentiator.
The most impressive candidates they'd interviewed had built a working RAG system on their own data — their college notes, a policy document, anything — and could explain what broke and how they fixed it.
Prompt engineering is now table stakes, not a differentiator. Every candidate knows to say 'you are an expert in X' and 'think step by step.' What separates people: systematic evaluation.
Can you measure whether a prompt is better? Can you build an eval harness that scores 50 outputs against a ground truth set? That meta-skill — treating prompt quality as a measurable engineering problem — is what startups mean when they say they want someone serious about AI.
One manager put it bluntly: 'I can teach prompting in a week. I cannot easily teach the instinct to test and measure. That's an engineering mindset, and either you have it or you don't.'
Production AI is expensive if you're not thinking about it. GPT-4 at scale can cost a startup thousands of dollars a month in ways that aren't obvious during development. Candidates who have thought about token budgets, caching strategies, and when to use smaller models are immediately more valuable.
Knowing when GPT-4o is overkill vs. when a smaller model like Mistral or Llama fails — and how to make that decision systematically — is a skill that matters the moment you're shipping something real.
One startup we spoke to specifically mentioned they'd passed on a candidate who had impressive ML theory knowledge but had never thought about cost optimization. 'We're not a research lab,' the hiring manager said. 'We care about unit economics.'
"The engineers getting hired at AI-forward startups aren't necessarily the ones who understand transformer architecture. They're the ones who can make AI work reliably in a product that real users depend on, without burning through the runway."
Career tactics, technical deep-dives, and honest advice — weekly.
