Meet Mr. Prompty! Break the Tasks Down and Chain of Thought: The Dynamic Duo of Prompt Engineering

Welcome back, prompt pioneers! I hope you've been practicing your prompt engineering skills since our last rendezvous. 

Today, we're diving deeper into the rabbit hole of prompts. We're going to tackle two techniques that are so important. They could be the Batman and Robin of prompt engineering: Break the task down and Chain-of-Thought.

Remember when you were a kid and your math teacher would insist that you 'show your work' on every problem? You might have thought they were just being difficult, but they were teaching you a valuable skill you've been using ever since, even if you didn't realize it. Like breaking down a math problem into smaller steps, making it easier to solve, the Break the task down and Chain-of-Thought techniques simplify complex tasks by breaking them into smaller, manageable parts. It's like taking a giant, scary monster of a problem and turning it into a bunch of cute, little mini-monsters that are much less intimidating to tackle.

So, if you ever thought that showing your work in math class was a waste of time, think again! You were being taught prompt engineering techniques even before it was invented! WOW :-)

System Prompt vs. User Prompt

Before we dive into our topic, let me clear up any confusion between the system prompt and user prompt because they are both essential components in human-computer interaction, particularly in natural language processing (NLP) and dialogue generation. 

System prompts are texts or instructions provided to the language model, guiding it to generate coherent and contextually relevant responses. These prompts act as the initial input to the AI system, setting the conversation's tone, context, and subject. System prompts are typically written from the perspective of the AI, indicating the desired behavior or role it should adopt during the interaction.

User prompts are the input provided by the user, representing their questions, requests, or statements during the conversation with the AI system. 

By combining system prompts and user prompts, AI systems can offer tailored and contextually relevant responses, enhancing the overall user experience in various applications, such as chatbots, virtual assistants, and dialogue-based language models.

Break the Task Down

As I said, the basic idea is to divide a more significant task into smaller, manageable subtasks. This technique can be advantageous when dealing with prompts requiring multiple steps to complete or containing many components.

Let's start with an example:

SYSTEM PROMPT:
You are a famous poet who wants to write a poem about a flower. 
You will be given instructions on how to complete the task.
        

USER PROMPT:
You will identify the main features of a flower, choose a flower 
to write about, brainstorm some ideas for the poem, write a draft, 
revise the poem, and publish the poem

===
Instructions:

- Identify the main features of a flower.
    - What are the different parts of a flower?
    - What are the colors of a flower?
    - What are the shapes of a flower?
- Choose a flower to write about.
    - What kind of flower do you want to write about?
    - Why did you choose this flower?
- Brainstorm some ideas for the poem.
    - What are some things you want to say about the flower?
    - What kind of poem do you want to write?
- Write a draft of the poem.
    - Start writing the poem.
    - Don't worry about making it perfect yet.
- Revise the poem.
    - Read the poem aloud.
    - Make changes to the poem.
- Publish the poem.
    - Share the poem with others.        

You can use this prompt to help an LLM to write a poem about a flower. The LLM can focus on each step individually and produce a better poem by breaking down the task into smaller, more manageable steps.

The user prompt can be used in a few different ways:

• You can send the full prompt in one shot.
• You can break it down into smaller chunks and send it to the LLM one step at a time.

Sending the entire prompt at once will prompt the LLM to start working on the task right away. However, if the task is too complex or lengthy, it may be unable to finish within the token limit. Alternatively, breaking the prompt into smaller segments allows the LLM to concentrate on each step separately, resulting in more successful task completion.

As you probably noticed, I have used other prompt approaches in the example of the poem because you can mix different approaches:

• Role-Playing: This is used in the system prompt "You are a famous poet who wants to write a poem about a flower." By instructing the AI that it has a specific role (in this case, a poet), we can help to guide its responses in a more targeted direction.
• Use of clear and concise language: The prompt is written in clear and concise language that is easy for the LLM to understand.
• Few-Shot Prompting: (discussed in our previous article) The prompt provides examples of how to complete each step. This helps the LLM to understand what is expected of it.
• Use of specific instructions: The prompt uses instructions that tell the LLM what to do. This helps the LLM to focus on the task and avoid getting distracted.
• Use of positive reinforcement: The prompt uses positive reinforcement to encourage the LLM to complete the task. This helps the LLM to stay motivated and engaged.

Chain-of-Thought

Chain-of-Thought is a different take on the 'break the task down' strategy. Rather than dividing a task into smaller parts, this approach guides the model to follow and display each step in the process. Doing this minimizes the chances of errors in the results and simplifies the evaluation of the model's response".

This particular technique can be studied in depth in this scientific paper.

The researchers tested this method on three large language models and found that it significantly improved their performance on various tasks, including arithmetic, commonsense reasoning, and symbolic reasoning. This method helped achieve the highest accuracy recorded on a challenging benchmark of math word problems.

Let's see with an example:

SYSTEM PROMPT:
You are an AI assistant that helps people find information.        

USER PROMPT:
Who was the first person to walk on the moon? 
Take a step-by-step approach in your response, cite sources, 
and give reasoning before sharing a final answer in the 
below format: ANSWER is: <name>        

And this is the answer from the AI:

Step 1: Determine the year the first person walked on the moon

  The first person to walk on the moon was Neil Armstrong, 
  who did so on July 20, 1969.
  
Step 2: Find a list of astronauts who were part of the Apollo 11 
mission.
  
  The Apollo 11 mission was the first mission to land humans on the 
  moon. The crew of the Apollo 11 mission consisted of Neil Armstrong, 
  Buzz Aldrin, and Michael Collins.
    
Step 3: Determine the first astronaut to walk on the moon.
    
  Neil Armstrong was the first astronaut to walk on the moon. 
  He stepped foot on the moon at 10:56 p.m. EDT on July 20, 1969.
    
Step 4: Cite sources.
    
  The official website of NASA is a reliable source for information 
  about the Apollo 11 mission. The website provides a list of the 
  crew members of the Apollo 11 mission, and it also includes a  
  transcript of Neil Armstrong's first steps on the moon.
    
Step 5: Give reasoning.
    
  Neil Armstrong was the first person to walk on the moon because 
  he was the commander of the Apollo 11 mission. 
  The space mission commander is responsible for leading the mission 
  and ensuring it is successful.
    
Therefore, the ANSWER is: Neil Armstrong.        

Break the task down vs. Chain-of Thought.

You might be curious about the practical differences between the two techniques. In the Break the task down technique, the user is responsible for breaking the task down into smaller, more manageable steps. The LLM then follows the user's instructions to complete the task.

In the Chain-of-thought technique, the LLM is responsible for breaking the task down into smaller steps. The LLM uses its knowledge of the world and its ability to reason. The LLM then generates a chain of thoughts that leads to the solution of the task.

The Break the task down technique is often used for well-defined tasks with a clear set of steps. The Chain-of-thought technique is often used for less well-defined tasks requiring more creativity. So, in essence, the key difference lies in who is responsible for breaking down the task: the user or the AI model.

Final Thoughts

I believe that the Chain-of-Thought paper highlights two crucial factors that are applicable to the entire field of prompt engineering: firstly, the acknowledgment of the significance of prompt engineering, and secondly, the importance of Commonsense Reasoning in AI.

(again) The importance of prompt engineering

In their research, the team discovered that only some have the same skill level when creating effective chain-of-thought prompts. Two co-authors could not develop prompts that effectively solved a task during a preliminary experiment. In contrast, another co-author created a chain of thought that flawlessly solved the same task. This implies that developing stronger chain-of-thought annotations could be a promising area for future research.

Commonsense Reasoning

Commonsense reasoning is a term used in artificial intelligence to describe the ability of a system to make assumptions or inferences about the world that humans typically understand without explicit instruction. It involves understanding and applying knowledge about everyday situations humans generally take for granted.

For example, if you read the sentence, John put the ice cream on the table and went to play. When he came back, the ice cream was gone. Commonsense reasoning allows you to infer that the ice cream likely melted because it was left out, even though this isn't explicitly stated.

In the context of AI, commonsense reasoning is a challenging problem because it requires a broad understanding of the world, including physical laws, societal norms, and general knowledge about people and objects. It's one of the areas where AI systems often need help because they need more experience to make these kinds of inferences.

In the paper, the researchers tested the chain-of-thought prompting method on tasks that required commonsense reasoning. They found that the method improved the performance of large language models on these tasks, suggesting that it could help enhance AI's ability to reason in a commonsense way.

Completing complex tasks effectively with LLMs can be mastered with the proper techniques. The key is finding the best technique for each task and LLM. Keep practicing with your prompt, and you'll soon be able to perform this magical art like a true master.

Happy prompting

M.