When is it ok to slop your colleagues?
Opportunities to beclown ourselves abound
Organizations are rapidly adopting AI but don’t have clear norms around when it’s acceptable to use it as a substitute for one’s own work. As these norms form, we’re in an exciting/spooky time with a high risk of doing silly things.1
My rule of thumb:
If quality matters, then I gotta be sure that I’m using AI to amplify, rather than replace, my own thinking
If slop is acceptable for the task, that’s totally fine - I just need to explicitly label it as such for my colleagues.
If you can’t independently verify the quality of the content, don’t send it to someone else without a disclaimer.
Two ways AI can be used are helping you with legwork for something you understand and doing your thinking for you.
For example, if I’m writing a research paper surveying benchmarks, I know how to review individual benchmarks to find representative samples and thus form a thesis. But because it would take a long time to do so, I outsource it to AI. I know what a good response looks like, and I have a prior on what the right answer should be, so I can guide the AI properly and know when to double-click on something that seems off. And because I know how to do the task, I can review the AI’s process and judge whether that workflow is likely to lead to good results.
By contrast, I am thoroughly unqualified to deeply comment on why one type of GPU is better suited for LLM inference than another. I could prompt AI to write such an essay, but I would have no ability to judge its quality. If I needed to produce such an essay, I would clearly label it as AI-generated, so my colleagues can apply the proper level of skepticism.
(And ideally, I would actually just link directly to the convo in which I generated it, so if the recipients want to tweak the prompt or fork the convo, they can do so.)
To be clear, sometimes directly-generated AI content is the best thing to do. Most meetings are adequately summarized by dumping a transcript into an LLM. I would just recommend labeling it as such.
I don’t wanna be a Claude Concierge
Many of us are looking over our shoulders, wondering how exposed we are to AI job loss.2 As an intuition pump, when I’m using AI, I ask myself:
If it were someone’s first day at my job, and they were doing the same task I’m doing right now, how much worse would their work product be than mine?
If the answer is “not at all”, then it suggests that my current workflow is basically being a white-glove prompt typist.
If the answer is “dramatically worse”, then it’s probably because I’m applying my own taste/judgment/context/experience.
The latter category is great, but the former category is at risk of evaporating.
For instance, imagine you’re a financial analyst. Your boss sends you a corporate earnings report, and you’re responsible for extracting some key financial metrics. Three years ago, you read the PDF yourself. Now you give the PDF to an AI, spot-check the results, and forward them to your boss.
Soon your boss will figure out how to use AI directly, and you’ll no longer be asked to do this.
So the question is: what value are you providing above what the AI is doing? How can you have an impact in your role that isn’t trivially replaceable by AI? Are you providing differentiated value, or are you just a temporary stopgap until AI finishes diffusing through your organization?
What should you do in this situation? It depends on the type of task.
If AI does a great job at the task today: teach your boss to use AI directly, then use the freed-up time to find work that actually needs you.
If it would be helpful to have the task done better: find a way to apply your unique judgment, context, or perspective – use the AI as an aid to your thinking, not an endpoint.
For the financial analyst example, this might look like going beyond extracting key financial metrics to telling a broader narrative about the facts that are most important to your org – something the AI won’t have a good sense of without a ton of context engineering.
What’s the best way to use AI as a collaborator, not a replacement?
When faced with the blank prompt box of a new conversation, there’s a temptation to just ask the AI to one-shot your end goal:
Write an investment committee recommendation for a potential investment in AcmeCorp at $821/share. Make no mistakes.
Instead, I recommend asking yourself: “keeping myself as the ultimate font of taste and judgment, what subtasks can I delegate to AI?”
For instance, when writing that investment committee memo, here are some ways to use AI as an assistant while preserving your own taste and judgment:
Critiquing. “Give me the strongest counter-argument against this thesis. Point out all gaps, ambiguities, and weak points.”
Brainstorming examples. “Give me examples of quotidian situations where people are apt to over-optimize”
Specific research questions. “Roughly when does it seem like Goodhart’s Law first became a mainstream idea? Check for the idea itself, even if it wasn’t coined Goodhart’s Law until later.”
Copy editing.
Tone shifting. “Change this paragraph to convey the same idea in a gentler way.”
Generating visualizations / images.
And, as you’re prompting, be sure to do so thoughtfully.
Imagine someone who is asked to plan activities for a team offsite. The person asks ChatGPT, then forwards the result to their team for review. But their prompt didn’t include the fact that the offsite is happening in June. By then, the weather will be totally different, which means that many of ChatGPT’s suggestions are useless. So they’ve asked their team to look at something that they themselves didn’t put much care into, thereby wasting the team’s attention.
AI has, in many instances, excused us from the burden of coming up with an answer. But it has not yet freed us from the necessity of asking the right questions.
Overcoming lack of domain expertise
It’s hard to get good AI output when you don’t know enough to judge for yourself if what it’s saying makes sense. But there are a few things you can do that are helpful at the margins:
Use the longest-thinking model you have access to. Within a size class, the frontier models cluster together – “how long it thinks” is a much bigger predictor of performance than “which specific model did you choose”.
Ask models to critique each other. Generate with ChatGPT Pro, ask Claude and Gemini to critique it, feed those critiques back into ChatGPT and ask it whether it agrees, etc. Keep doing this until the models converge or you’re convinced they won’t. This doesn’t guarantee a high-quality response, but it helps. Even if you can’t assess the actual subject matter merits of an argument, you can sometimes get a vibe on rigor just by observing the structure of arguments being made.
Meta-prompt. Tell a model what you’re trying to accomplish, and ask it to write the prompt for you. It’ll produce a ton of detail for you to react to – you can emphasize certain points, clarify others, etc. Ask the model what you’re missing.
Best-of-n. Try the same prompt with multiple models, and/or the same model multiple times. Review the key differences (which you can also use a model to help with). Do the differences seem like reasonable disagreement, or do they reveal where the models are speaking overconfidently?
The optimal number of mistakes is not zero
If your takeaway from this piece is “sweet, new anxieties unlocked,” that’s the wrong takeaway. We’re only going to learn how to use AI well by using it, getting some of it wrong, and adjusting quickly. I’d rather work with someone who occasionally misjudges the line than someone who never gets close enough to find it.
The entire basis of my opinions in this post is what I’ve learned from my own mistakes. We can’t find the line without taking the risk of crossing it!
This post focuses on the epistemic angle of using AI. Out of scope but also important is making sure that you have permission to be sharing the data in your prompts with AI.
Then there’s the amusing archetype of AI job impact predictions, where every job is at risk, except the job held by the person making the prediction.
On the topic of "slop your colleagues" the only question worth asking is: "What would Richard Paul do?"