Sample interview
Applied AI Engineer
7 questions · 60 minutes
Production LLM features: retrieval, tools, evaluation and safety.
Illustrative: the job post, candidate and results are examples.
Why this interview
Requirements
The job requires
- Retrieval (RAG)
- Evaluation
- Tool use
- AI security
- Production LLM features
Claims
The résumé claims
- Built a RAG support assistant
- OpenAI and Anthropic APIs
- LangChain
Focus
So the interview tests
- Retrieval quality
- Debugging RAG failures
- Prompt injection
- Evaluation design
The plan we propose
Every question comes with the reason it was chosen. The interviewer can swap any of them before sending.
- 1
Retrieve an exact error code and its meaning
Select all · Advanced · ~5 min
Why this question: The résumé claims a RAG assistant. Tests whether retrieval design goes beyond embeddings.
- 2
Diagnose an unsupported RAG answer
Ordering · Senior · ~6 min
Why this question: Separating corpus, retrieval and generation failures is the core RAG debugging skill.
Try it below - 3
Contain prompt injection from retrieval
Select all · Senior · ~6 min
Why this question: The job requires AI security. Defenses must hold even when the model follows injected text.
Try it below - 4
Design a narrow tool schema
Select all · Advanced · ~5 min
Why this question: Tool use: limiting what a model can do through the tool interface itself.
- 5
Build an AI regression set
Select all · Advanced · ~6 min
Why this question: The job requires evaluation. Tests whether the candidate can build a set that catches regressions.
- 6
Validate model-generated actions
Select all · Advanced · ~5 min
Why this question: Model output that drives actions has to be treated as untrusted input.
- 7
Design model-provider failover
Select all · Senior · ~6 min
Why this question: Production LLM features need bounded, observable provider failover.
Try two of the questions
Answer as the candidate would. The interviewer guide beside each question is what the interviewer sees.
Diagnose an unsupported RAG answer
Candidate view
Exactly what the candidate sees. Try answering it.
A support assistant gives a confident answer that the documentation it cites does not actually support. The team suspects the prompt and wants to rewrite it.
Before anyone changes the prompt, order the steps you would take to find which stage failed and keep it from happening again.
Drag the steps into order, or use the arrows.
- 1Test whether the generator uses the retrieved evidence and abstains when it is insufficient.
- 2Add the failure to a labeled regression set for the responsible stage.
- 3Check whether the answerable source exists in the indexed corpus.
- 4Inspect retrieval candidates, filters, ranks, and chunk boundaries.
Ready when you are.
Contain prompt injection from retrieval
Candidate view
Exactly what the candidate sees. Try answering it.
A RAG agent retrieves a document that tells the model to send customer data to an external URL. Select the defenses that remain effective even if the model follows part of the injected text.
Select every statement that applies
Choose an answer to check it.
See what the interview actually established
Built only from the interviewer's judgment on each question. The job and résumé decide what gets tested; they never count as evidence.
What the interview established
Strongest evidence was in AI security, Production AI systems and Retrieval & grounding. AI evaluation and Tool use & agents showed mixed evidence.
5 of 5 required areas evaluated · 7 questions judged by the interviewer
Required competencies
From the job. Open a row for the questions and notes behind it.
AI evaluationMixed
Diagnose an unsupported RAG answer
StrongInterviewer
Checked corpus, then retrieval, then generation before touching the prompt.
Build an AI regression set
WeakInterviewer
Built the evaluation set only from cases the current system already handles.
Tool use & agentsMixed
Design a narrow tool schema
StrongInterviewer
Narrowed the schema and re-validated arguments in trusted code.
Validate model-generated actions
MixedInterviewer
Validated structure but forgot business authorization until asked.
AI securityDemonstrated beyond résumé claimsStrong
Contain prompt injection from retrieval
StrongInterviewer
Put capability limits and approvals outside the model.
Design a narrow tool schema
StrongInterviewer
Narrowed the schema and re-validated arguments in trusted code.
Production AI systemsDemonstrated beyond résumé claimsStrong
Design model-provider failover
StrongInterviewer
Bounded failover with a shared budget and recorded which model answered.
Retrieval & groundingStrong
Retrieve an exact error code and its meaning
StrongInterviewer
Combined keyword and semantic retrieval with reranking, and planned to measure recall.
Diagnose an unsupported RAG answer
StrongInterviewer
Checked corpus, then retrieval, then generation before touching the prompt.
Contain prompt injection from retrieval
StrongInterviewer
Put capability limits and approvals outside the model.
Also observed
Not required for this role, but the questions produced evidence.
Reliability & incidentsStrong
Design model-provider failover
StrongInterviewer
Bounded failover with a shared budget and recorded which model answered.
Claimed on the résumé but not tested: Prompt & context design.
How they worked
The kind of work the questions asked for, across competencies.
Build interviews like this for your roles.
Post a job, and we propose an interview for every applicant from the job and their résumé.