1 week Ago
Learn about three new best practices for composing prompts and being a great prompt engineer. In today’s column, I will explain three new best practices for coping with prompt wording sensitivities when using generative AI and large language models (LLMs). The deal is this.
It is widely known that you must word your prompts cautiously to ensure that AI gets the drift of what you are asking to do. Sometimes, just a meager word or two can radically change what the AI interprets your question or instruction to consist of. Generative AI can be hypersensitive to what you say in your prompts.
It is often a touch-and-go proposition. This can be exasperating. Very exasperating.
Plus, there is a potential cost involved. Namely, if you are paying to use generative AI, you’ll be getting an off-target response, for which you are paying regardless of whether the AI grasped your intention or not. As they say, all sales are final.
The same goes for snubbed or misunderstood prompts. Savvy prompt engineers know that they must mindfully word their prompts. Casual users typically catch onto this consideration after a considerable amount of muddling around, along with lots of trial and error.
No worries, I’ve got your back and will provide straight-ahead advice on what you should be doing with your prompts. My inspiration is due to a recently posted research study that sought to empirically unpack the prompt sensitivities associated with several generative AI and LLM apps. The good news is that a set of best practices can be squeezed or distilled from the results.
I will show you my distillation of three prompting rules or techniques that you can add to your prompting skillset and toolkit. Trump’s Cabinet: Here Are His Picks For Key Roles—RFK Jr, Doug Burgum, Matt Gaetz And More What Time Is Netflix’s Live Jake Paul Vs. Mike Tyson Fight On Friday? NYT ‘Strands’ Today: Hints, Spangram And Answers For Friday, November 15th Let’s talk about it.
This analysis of an innovative proposition is part of my ongoing Forbes.com column coverage on the latest in AI including identifying and explaining various impactful AI complexities (see the link here ). For those of you specifically interested in prompting and prompt engineering, you might want to take a see my comprehensive analysis and description of over fifty notable prompting techniques at the link here .
My Proffered Best Practices On This Topic I will jump right into my three key recommendations. After doing so, I then cover some examples so that you can tangibly see what goes on and what to do. Finally, in the end, I’ll do a rundown of the research study that inspired these bits of sage advice.
One quick point. Unlike a lot of the prompt engineering techniques that rely upon a certain phrasing or catchphrase, such as invoking chain-of-thought or CoT via telling the AI to work on a stepwise basis (see for example the link here ), these tips or rules are broader and not a specific set of commands or wordings. In a sense, these are about how to fish and the ways to use a rod and reel.
I’m sure you’ll be able to mentally convert these into a handy habit of ways to suitably word your prompts. That also applies to avoiding the gotchas of less amenable prompts. Here are my three recommendations: I shall tackle each one, one at a time.
#1: Best Practice That Larger AI Gets The Drift As noted above, the first rule is this: A state-of-the-art prompt engineer realizes that larger AI models tend to be better at prompt interpretation. Larger LLMs often do a better job of interpreting user prompts, thus, this makes life easier for users since the generative AI can more readily get the drift of what the user wants or is asking. If you end up using a lesser-sized AI, you’ll need to be especially mindful of how you carefully word your prompts.
When switching among different AI models, find out whether the AI is in the larger category or the less large category, so you’ll know how careful you need to be with your prompts. You can rest somewhat easier with larger AI as to wording your prompts. Even the barest prompt will potentially hit home.
With the lesser-sized AI, you need to really watch your wording and be especially precise. In that sense, it is easier to shift to a larger AI since you already are used to wording things precisely and can either continue doing so or opt to be loose. Those that drop down to a lesser-sized AI will need to wake up and start being precise again, assuming they got accustomed to the looseness of the larger-sized AI.
Why are larger generative AI apps better at interpreting prompts? The answer is a mixture of having a larger base of data training and a more extensive pattern-matching structure. They’ve seen it all, in a manner of speaking, and can parse your prompts with greater intensity. I’m not saying that a larger AI is always better at this.
There are certainly going to be times at which this doesn’t pan out. By and large, it’s a reasonable rule of thumb and worth knowing. I went ahead and came out with a prompt that shows the difference between interpretation by a larger AI and a lesser-sized AI.
Here we go. Take a close look at the two responses. The response by the lesser-sized AI is generally apt, but not nearly as on-target as the response by the larger AI.
The larger AI got my drift that I wanted creative ideas. The lesser-sized AI seemed to miss that component of my prompt. #2: Best Practice That Data Training Is King You might recall that the second of my best practices said this: A state-of-the-art prompt engineer realizes that if an LLM has more knowledge on a particular topic that pertains to the topic of the prompt, the AI will almost surely do a better job of interpreting the meaning of the prompt and tend to provide a better response accordingly (all else being equal).
Allow me to elaborate. Suppose you aim to ask a question about an obscure type of snail that normally isn’t discussed in everyday literature on such creatures. Most of the generic generative AI apps are data-trained across the board and not versed in narrow niches.
The odds of generic generative AI having covered that rare snail are probably low. There are newly emerging generative AI apps that are gradually being augmented with specialized niche data, or that are built from day one with such data. Thus, a generative AI app that is devised for say biologists is likely to have content about the uncommon snail.
The gist is that if the AI you are using is relatively versed in the question or matter you are asking about, the capability of interpreting your prompt tends to go up. The AI has a basis for getting the drift. In contrast, an AI that doesn’t have data training on the given topic will likely flail around to figure out what in the world you are asking about.
Please note that this does not necessarily have anything to do with the size of the LLM. It is more what the LLM has been data trained on. When using generative AI, find out what data training the AI has had on the topics you want to ask about.
If the generative AI isn’t versed in those topics, you’ll need to go all out in your prompts to specify exactly what you want. Here is my example of this phenomenon. The generative AI app that wasn’t versed in the topic at hand was reaching to answer and wasn’t even fully on-target to the posed question.
You might observe that the AI versed in the topic had a more pertinent vocabulary, such as noting the importance of cover cropping and no-till farming. The response by the less-versed AI is not incorrect, it is just unable to grasp the totality of the question being asked. The response suffers accordingly.
#3: Use An Example If You Can Come Up With One The third of my stated best practices said this: A state-of-the-art prompt engineer realizes that if you give an LLM even just one example in a prompt, suggesting what the user intends, this makes a significant difference regarding how well the AI responds to the prompt. Whoa, some snarky prompting pros are undoubtedly exhorting, everyone knows or ought to know that by including examples in a prompt you are going to likely get a better interpretation and better results (see my coverage of using examples in prompts, at the link here ). Sure, I know you know that.
First, not everyone does. Secondly, some overly zealous pros proclaim that the user must enter several examples. According to the latest research, one example will often do.
Yes, just one example, known as a one-shot or single-shot approach. I want to clarify that I am not saying you should not provide multiple examples. Nope, I didn’t say that.
I am merely emphasizing that even just one example can make a whale of a difference. I guess it’s like buying a lottery ticket — if you don’t get one then you aren’t in the game. Once you’ve got one, you can get more, which might increase your odds, but sometimes the added odds aren’t worth the extra effort.
If you have more than one example handily available, sure, try undertaking a few-shot or multi-shot approach of listing them all. But do so mindfully. Things can readily veer into outer space with multi-shot examples.
For example, two major disconcerting problems often occur. I see this lamentedly happen all the time while instructing classes on prompt engineering and guiding attendees to avoid prior adverse practices that they’ve formed as bad habits. The first qualm is that you accidentally overload your prompt with too many examples, and it gets confusingly bloated.
The AI can’t make heads or tails of what you are saying. Secondly, and worse still entails giving examples that aren’t relevant. This can throw the AI for a loop and cause it to land in some oddball realm that no longer matches your topic at hand.
Whew, that was a lot. Time to showcase an example. Let’s explore a prompt that doesn’t have an included example and then take another attempt but with an included example.
Here we go. Next, take a look at what happened when I tried again and include an example in the prompt. I assume you can plainly see that the prompt that included an example fervently spurred a more extensive answer.
The prompt that lacked an example was probably too sparse for the AI. The AI was unable to latch onto what I wanted. My prompt should have been more descriptive.
A means to be descriptive is by including an example. The example readily clued in the AI about what I was seeking. Leading Research On Prompt Sensitivity A recently posted study caught my eye in terms of some of the latest work on prompt sensitivities, in a paper entitled “ProSA: Assessing And Understanding The Prompt Sensitivity Of LLMs” by Jingming Zhuo, Songyang Zhang, Xinyu Fang, Haodong Duan, Dahua Lin, Kai Chen, arVix, October 16, 2024, which made these salient points (excerpts): You are encouraged to read their full study to garner more nuances about dealing with prompt sensitivities.
As I mentioned at the outset, my three best practices are from my own experience and aren’t necessarily codified in the same way in the research study. I took a snippet of salt and pepper, added some additional spices, and came up with a distinctive meal that I hoped would be of satisfaction to you. Where To Go Next With This My follow-up assignment for you is to take a few moments, if you can, and try to test out these best practices in whichever generative AI you are currently using.
Here’s why. The perhaps most important of all rules of thumb about prompt engineering consists of three crucial words, consisting of practice, practice, practice. So, please try practicing the matters covered here.
One last remark before we end today’s discussion. As much as possible, I prefer to adopt prompting techniques that have been empirically explored. We can all come up with seat-of-the-pants ideas about good prompts versus bad prompts.
Leveraging research is a valuable way to feel at least modestly secure that your ad hoc sense of prompting seems to be reasonably well-grounded. That’s what I always try to do. It is an aspirational goal.
Okay, the final thought for now is based on a favorite quote from the popular but rather unnerving movie Training Day , by a rambunctious character played by famed actor Denzel Washington, in which he utters an immortal line: “It's not what you know; it's what you can prove.” To some degree, I urge prompt engineers to strive in that direction, whenever feasible..
