How to evaluate an AI developer when you're not technical

Evaluating an AI developer is genuinely hard without an ML background. The vocabulary sounds the same from everyone. GitHub profiles look plausible. And the difference between someone who can build and maintain a production ML system versus someone who has done a few Kaggle competitions and watched some YouTube tutorials isn’t obvious from a resume.

Here’s how to close that gap without becoming an ML expert yourself.

This covers individual developers and freelancers specifically. If you’re evaluating agencies rather than individuals, the AI development red flags guide covers what to look for at the agency level.

Start with the work, not the CV

Most AI developers have GitHub repos, Kaggle profiles, or links to live projects. What you’re trying to understand isn’t what they built. It’s whether it survived contact with reality.

Ask to see something they built that went into production and was used by real people for more than a few weeks. Demos and side projects tell you someone can get to 80%. Production systems tell you they can get to 100 and keep it there. That’s a different skill, and it’s the one that matters.

When they walk you through a project, listen for how they talk about failure:

• Did the model fall short of expectations initially, and what did they do about it?
• Were there data issues mid-project, and how were they handled?
• Did they set up monitoring? What happened when performance started degrading?

People who have shipped production ML systems talk about this stuff naturally, because they’ve had to deal with it. People who haven’t tend to describe their work in terms of what they built, not what they learned when it broke.

Give them a small paid trial

The most reliable signal, by a significant margin, is a small piece of real work paid at their rate. It doesn’t need to be a big commitment: a few hours on a well-defined problem is enough to see how someone actually thinks.

Pick something close to your real project. If you’re building a document classification system, give them a sample of your data and ask them to sketch an approach and put together a rough prototype.

Pay attention to what they do before they start coding:

• Do they ask what accuracy you actually need?
• Do they ask what happens when the model is wrong?
• Do they ask whether you have labelled examples or would need to start from scratch?
• Do they clarify how the result will be used before designing the approach?

The questions someone asks before starting tell you more than the work itself. A developer who just starts building, without probing the problem first, is not someone you want making architectural decisions on a real project.

Ask about failure, not success

Most interviews are structured to surface strengths. For evaluating AI developers without technical depth yourself, questions about failure are more useful.

“Tell me about a model that didn’t work as well as you expected. What did you try?”

A good answer tells you more about actual competence than any description of someone’s best project. Weak candidates struggle to name something specific. Strong ones have several examples and can tell you precisely what went wrong and why.

“What would you monitor after this goes to production?”

If they mention accuracy on the training dataset, that’s a red flag. That’s not what you monitor in production. If they talk about live prediction distributions, flagging low-confidence outputs, alerts for data drift, retraining schedules, that’s someone who has thought about the full lifecycle. Anthropic’s model specification and Google’s ML Engineering practices guide both treat monitoring as a first-class concern in production AI.

“What’s the hardest data problem you’ve run into?”

The answer tells you whether they’ve spent real time in the messy part of the work, which is where most of the time actually goes.

Watch how they handle uncertainty

AI work has more genuine uncertainty in it than most software engineering. Before seeing your data, a developer can’t tell you exactly what accuracy is achievable. Anyone who speaks in confident numbers at the first meeting is either working from deep experience with your exact problem type (possible) or overselling (more likely).

What you want is calibrated honesty. Can they give you a reasonable range and explain what would change it? Can they explain a trade-off clearly without dumbing it down?

Watch out for jargon used to avoid saying anything concrete. Compare:

• “We’ll leverage state-of-the-art transformer architectures to achieve optimal performance.” This means nothing.
• “Given the size of your labelled dataset, I’d expect somewhere around 85% accuracy with a fine-tuned BERT model, but I’d want to verify that against your actual data in the first week.” This is useful.

If you can’t follow their explanation of something that directly affects your project, that’s going to be a problem for the whole engagement.

References are worth calling

Freelance AI developer references get skipped constantly. Two questions worth asking:

• What did they underestimate about the project?
• Would you give them access to your production systems again?

The first gets past the diplomatic surface-level positive. Everyone says their former developer was “great to work with.” Fewer can name something specific they misjudged and how they handled it. The second is harder to answer diplomatically if the honest answer is no, which makes it more useful than “would you recommend them.”

What doesn’t actually tell you much

Years of experience is a weak signal in this field specifically. Things have changed fast enough that four years of focused work in the right areas can mean more than ten years of adjacent work. Focus on what they’ve shipped.

Certifications are mostly irrelevant. There are strong developers with no credentials and weak ones with several.

Confidence in initial meetings tells you how someone performs in initial meetings, which is not the same as how they perform when a model is underperforming at month three and the client is asking questions.

The strongest filter is still paid work on a real problem, even a small one. Everything else is an approximation of that.

Useful places to find AI developers

If you’re sourcing rather than evaluating:

• Toptal: pre-vetted senior ML engineers, higher rates
• Arc.dev: senior engineers with remote-first experience, good vetting
• Upwork: broad marketplace, higher variance, requires careful screening
• Andela: pre-vetted engineers from African tech markets, staff augmentation model
• Hugging Face community: for open-source practitioners, good for identifying people who are active in the ML community

Also worth checking: LinkedIn filtered by ML frameworks (PyTorch, TensorFlow, Hugging Face), and GitHub profiles of contributors to open-source ML projects. Activity in these communities is a real signal.

Frequently asked questions

What’s the difference between a data scientist and an AI developer?

Broadly: data scientists focus on analysis, modelling, and insights, often working in notebooks and producing reports or model experiments. AI developers (or ML engineers) focus on building production systems: the code, infrastructure, deployment, and monitoring that makes a model run reliably at scale. The roles overlap significantly, but if you need something deployed and maintained in production, you want someone with ML engineering depth, not just modelling skill.

How many hours should a paid trial be?

Three to eight hours is usually enough to see how someone approaches a problem. More than that starts to feel like you’re extracting free work; less than three hours doesn’t give them enough time to show their process. Define the scope clearly in advance, set a deliverable (even just a written approach document and a rough prototype), and pay the full rate.

What frameworks should an AI developer know?

Depends on what you’re building. For most modern AI work: Python is non-negotiable, PyTorch or TensorFlow for model work, Hugging Face Transformers for LLM-related tasks, and at least one cloud ML platform (AWS SageMaker, Google Vertex AI, Azure ML). For LLM applications specifically: LangChain or LlamaIndex for orchestration, familiarity with RAG architectures, and experience with prompt engineering and evaluation. Ask them to walk you through their typical stack for the type of problem you’re solving.

Should I hire one AI developer or a team?

For most initial projects, one strong ML engineer plus one data engineer covers the essential skills. A solo developer handling everything (modelling, pipelines, deployment, monitoring) is a risk, both because the breadth is significant and because there’s no peer review. If you’re hiring through an agency, this is handled for you. If you’re building a freelance arrangement, consider whether you need at least two people who can review each other’s work. More on this in the in-house vs. agency vs. freelancer comparison.

How do I evaluate AI developer portfolios without being technical?

Look for specificity over impressiveness. Does the project description mention the problem being solved, the data used, the accuracy achieved, and what happened after launch? Or does it just say “built an ML model using Python and TensorFlow”? Ask them to explain one project to you as if you’re the business stakeholder who commissioned it. How they frame the work for a non-technical audience tells you a lot about how they’ll communicate throughout an engagement.