Shopify Shop App Agent Annotator

About this role

## About the role Shopify's **Shop App Agent** is the AI assistant buyers talk to inside the Shop app — product discovery and search, orders, shipping and tracking, account questions, and Shop Cash. When a shopper is unhappy with one of its answers, they tap thumbs-down — and that's all they leave. No comment, no reason, no survey. Your job is to work out *why*, from the transcript alone. ## What you'll do - Read a full shopper ↔ agent conversation, including the tool calls the agent made and what came back - Find the specific turn that received the thumbs-down - Classify it with one primary tag — the *domain* of the complaint: a disliked recommendation, a failed order lookup, an action the agent couldn't take, a rejection of AI itself, or a genuinely broken response — and one specific secondary tag within it - Write a short comment: what the shopper wanted, what the agent did or failed to do, and why those tags fit - **Commitment:** ~5 hours/week ## What makes someone good at this The hardest part is not guessing. Plenty of thumbs-downs have no visible cause — the agent did nothing wrong, or the shopper simply didn't like the answer. There is an explicit `unknown` label for exactly that, and using it honestly matters more than producing a confident-sounding reason. We would rather record "we don't know why" than invent a cause that sends the wrong signal to the model. ## Qualifications **Required:** - **Rule discipline** — you can apply a fixed taxonomy the same way across hundreds of conversations, and you notice when a case sits between two labels rather than forcing it. - **Comfort reading structured traces** — tool calls, their inputs, and their raw outputs, so you can tell what the agent actually did from what it merely claimed. - **Restraint under ambiguity** — you are willing to label something `unknown` rather than reach for a plausible-sounding cause. - **Clear, concise written English** — enough to explain in two or three sentences why a conversation got the labels you gave it. **Strongly preferred:** - **Consumer e-commerce fluency** — orders, tracking, returns, refunds, and a working sense of where a merchant's responsibility ends and the platform's begins. - **Prior annotation, labelling, or model-evaluation work** against a defined rubric or taxonomy. **Preferred (nice to have — we'll ramp you on the specifics):** - Hands-on experience with the Shop app or other AI shopping assistants as a shopper. - Familiarity with customer-support operations and escalation paths. ## What this is not This is not a customer-support role and not an engineering role. You are not fixing the agent and not replying to shoppers — you are diagnosing why a real shopper was unhappy, precisely and repeatably, so the team can measure where the agent falls short.

Skills & domains

• ai-training
• rlhf
• sme
• annotation
• Miscellaneous