Evaluating Prompt Engineering Skills: Practical Work Sample Activities

In the rapidly evolving landscape of artificial intelligence, the ability to effectively communicate with large language models and other generative AI tools has become a critical skill. This is the essence of prompt engineering – the art and science of crafting inputs that yield desired outputs from AI models. As organizations increasingly integrate AI into their workflows, the demand for individuals who can expertly guide these powerful tools is skyrocketing. However, evaluating this nuanced skill through traditional interview methods can be challenging.

Prompt engineering is far more than just typing a query; it involves a deep understanding of how AI models process information, an ability to structure requests clearly and unambiguously, and the iterative process of refining prompts based on observed outputs. It requires creativity, logical thinking, and a knack for anticipating potential misunderstandings by the AI. A skilled prompt engineer can unlock the full potential of AI, transforming it from a simple tool into a powerful collaborator that drives efficiency and innovation.

Given the practical nature of prompt engineering, theoretical questions or discussions about concepts are insufficient to truly gauge a candidate's proficiency. The best way to assess a candidate's ability to excel in this role is through realistic work samples and technical evaluations. These exercises provide a hands-on demonstration of their skills, allowing you to observe their process, problem-solving approach, and ability to deliver tangible results using AI tools.

Work samples for prompt engineering should simulate real-world scenarios the candidate would encounter on the job. This could involve generating specific types of content, extracting structured information from unstructured text, or guiding an AI to perform complex reasoning tasks. By observing how candidates approach these challenges, refine their prompts, and analyze the resulting outputs, you gain invaluable insight into their practical capabilities and potential for success in a prompt engineering role. The following activities are designed to provide just such a window into a candidate's prompt engineering prowess.

Activity #1: Text Generation Refinement

This activity assesses a candidate's ability to refine prompts iteratively to achieve a specific, high-quality text output. It demonstrates their understanding of prompt clarity, constraint handling, and the iterative nature of working with AI models.

Directions for the Company:

• Provide the candidate with a specific text generation task (e.g., "Write a concise summary of the attached article," or "Draft a short, engaging social media post about [topic]").
• Provide an initial, poorly written or overly simplistic prompt for that task, along with the unsatisfactory output it generated from a specific AI model (specify which model, e.g., "Using Model X").
• Provide the candidate with access to the AI model or a simulated environment where they can test their prompts.
• Set a time limit for the exercise (e.g., 30-45 minutes).

Directions for the Candidate:

• You will be given a text generation task, an initial prompt, and the output generated by an AI model using that prompt.
• Your goal is to refine the provided prompt through multiple iterations to produce a significantly better output that meets the requirements of the task.
• Explain your thought process and the changes you make in each iteration. Why did you make those specific changes? What were you trying to achieve?
• Present your final prompt and the best output you were able to generate within the time limit.

Feedback Mechanism:

• Instruct the interviewer to provide feedback on the candidate's initial approach and prompt refinement strategy.
• Provide specific feedback on one aspect of their prompt or process that could be improved (e.g., "Consider adding negative constraints," or "Try breaking down the request into smaller steps").
• Give the candidate 5-10 minutes to make one final adjustment to their prompt based on the feedback and generate a revised output.

Activity #2: Constraint-Based Text Prompting

This activity evaluates the candidate's skill in incorporating specific constraints and requirements into a prompt, including handling negative constraints (things the output should not include). This is crucial for controlling AI behavior and ensuring outputs meet precise specifications.

Directions for the Company:

• Provide a text generation task with several explicit positive and negative constraints (e.g., "Write a product description for a new gadget. It must be under 150 words, highlight feature X and benefit Y, use a friendly tone, and must not mention competitor Z or use jargon.").
• Specify the target AI model.
• Provide access to the AI model or a simulated environment.
• Set a time limit (e.g., 20-30 minutes).

Directions for the Candidate:

• You will receive a text generation task with a list of requirements and constraints, including things the output should explicitly avoid.
• Craft a single prompt designed to guide the AI model to produce text that satisfies all the positive and negative constraints.
• Explain how your prompt addresses each constraint.
• Present your prompt and the resulting output from the AI model.

Feedback Mechanism:

• Instruct the interviewer to provide feedback on how well the prompt incorporated the constraints and the quality of the resulting output.
• Provide specific feedback on one constraint that was not fully met or could have been handled more effectively in the prompt.
• Give the candidate 5-10 minutes to revise their prompt based on the feedback and generate a new output.

Activity #3: Complex Task Decomposition and Prompt Planning

This activity assesses the candidate's ability to break down a complex problem into smaller, manageable steps and plan a sequence of prompts (or a single, structured prompt) to achieve the final outcome. This tests strategic thinking and understanding of multi-turn interactions or advanced prompting techniques like Chain-of-Thought.

Directions for the Company:

• Provide a complex task that requires multiple steps or different types of AI output (e.g., "Analyze the key findings from this research paper [provide paper link or text], summarize them for a non-technical audience, and then draft three potential headlines for a blog post based on the summary.").
• Specify the target AI model(s) if multiple are needed, or confirm it can be done with one.
• Provide the necessary source material (e.g., research paper text/link).
• Set a time limit for planning and initial execution (e.g., 45-60 minutes).

Directions for the Candidate:

• You will be given a complex task requiring multiple stages of processing or output from an AI.
• Outline your plan for tackling this task using AI prompts. This plan should detail the steps you will take, the type of prompt you will use for each step, and how you will use the output from one step as input for the next (if applicable).
• Execute your plan by writing and running the necessary prompts.
• Present your plan, the prompts used, and the final output(s).

Feedback Mechanism:

• Instruct the interviewer to provide feedback on the clarity and logic of the candidate's plan and the effectiveness of their prompt sequence.
• Provide specific feedback on one area where the plan could be more efficient or the prompts could be better structured for the complex task.
• Give the candidate 5-10 minutes to explain how they would adjust their plan or prompts based on the feedback.

Activity #4: Persona and Style Adaptation Prompting

This activity tests the candidate's skill in guiding an AI to adopt a specific writing style, tone, or persona. This is valuable for content creation, marketing, and communication roles where brand voice and audience adaptation are important.

Directions for the Company:

• Provide a piece of source text (e.g., a technical explanation, a press release, a product feature list).
• Provide a clear description of a target persona or writing style the AI should adopt when rewriting the source text (e.g., "Rewrite this explanation for a 10-year-old," or "Adapt this press release into a casual, humorous social media thread," or "Rewrite this feature list as bullet points in the style of a formal business report.").
• Specify the target AI model.
• Provide access to the AI model or a simulated environment.
• Set a time limit (e.g., 20-30 minutes).

Directions for the Candidate:

• You will receive source text and a description of a target persona or writing style.
• Craft a prompt that instructs the AI model to rewrite the source text while fully embodying the specified persona or style.
• Explain how your prompt guides the AI towards the desired tone and style.
• Present your prompt and the resulting adapted text.

Feedback Mechanism:

• Instruct the interviewer to provide feedback on how well the generated text matches the target persona/style and the effectiveness of the prompt in achieving this.
• Provide specific feedback on one aspect of the prompt or output that could be improved to better capture the desired style or persona.
• Give the candidate 5-10 minutes to revise their prompt based on the feedback and generate a new output.

Frequently Asked Questions

Q: Which AI model should we use for these exercises?

A: It's best to use the AI model that the candidate will primarily work with in the role, if possible. Consistency is key for comparison across candidates. If the specific model is proprietary or unavailable, choose a widely available, capable model (like a recent version of GPT, Claude, or Gemini) and use the same one for all candidates.

Q: How strictly should we time the exercises?

A: Timing helps simulate real-world pressure and assesses efficiency. Stick to the suggested times, but allow a few extra minutes if a candidate is clearly on the verge of a breakthrough. The goal is to see their process under reasonable constraints, not to penalize slightly exceeding a limit.

Q: What if the AI generates unexpected or poor output despite a good prompt?

A: This is a valuable part of the evaluation! Prompt engineering isn't just about writing the perfect prompt once; it's about iterating and troubleshooting. Observe how the candidate reacts. Do they blame the AI, or do they analyze the output and adjust their prompt? Their ability to debug and refine is a key skill. The feedback mechanism is designed to test their coachability in this scenario.

Q: How do we evaluate the quality of the prompt and the output objectively?

A: Define clear criteria beforehand based on the task and constraints. For text generation, evaluate clarity, conciseness, adherence to constraints (word count, topics to avoid), tone, and overall quality relative to the task goal. For planning, evaluate the logic, efficiency, and completeness of the steps. Use a structured scorecard to rate candidates on these specific criteria for each activity.

Q: Can candidates use external resources or search the web during the exercise?

A: For most prompt engineering roles, accessing documentation or searching for prompt examples is realistic. Clarify your policy beforehand. Allowing access can show their resourcefulness, while restricting it tests their foundational understanding and creativity under pressure. For these specific exercises, providing necessary context (like the research paper in Activity 3) is sufficient; general web access can be permitted if it aligns with the role's expectations.

Q: What if a candidate is unfamiliar with the specific AI model we use?

A: If possible, inform candidates which model will be used ahead of time so they can familiarize themselves. If not, provide a brief overview of the model's general characteristics (e.g., good at creative writing, strong reasoning abilities, sensitive to negative constraints) before the exercise begins. Focus evaluation on their prompting principles and iterative process rather than model-specific syntax they couldn't know.

Hiring the best prompt engineers requires moving beyond theoretical discussions to practical, hands-on evaluation. By implementing structured work samples like the ones outlined here, you gain invaluable insight into a candidate's ability to effectively wield AI tools, a skill increasingly vital for success. These exercises, combined with a rigorous interview process, will help you identify candidates who can truly unlock the potential of generative AI for your organization. Yardstick's AI-powered tools can further enhance your hiring process, from creating precise AI job descriptions and generating relevant AI interview questions to building complete AI interview guides tailored to roles like Prompt Engineer.

Build a full interview guide for the Prompt Engineering role by signing up for a free Yardstick account: https://yardstick.team/sign-up