What is Temperature in Prompt Engineering?

Introduction

Prompt engineering is key to dealing with large language models (LLMs) such as GPT-4. “Temperature,” one of the most important prompt engineering parameters, greatly impacts the model’s behavior and output. This article examines the idea of temperature in prompt engineering, defines it, outlines its operation, and provides practical advice on utilizing it to modify an AI model’s responses.

Overview

• Introduction to Prompt Engineering: Understanding the importance of “temperature” in managing the behavior and output of large language models like GPT-4.
• Defining Temperature: Temperature regulates the randomness of a language model’s outputs, balancing creativity and determinism.
• Temperature Mechanics: It modifies the probability distribution of predictions, with lower values favoring high-probability words and higher values increasing output diversity.
• Practical Applications: Low temperatures are ideal for precise tasks, medium for balanced creativity, and high for imaginative outputs.
• Best Practices: Experiment with different temperatures, consider context, combine with other parameters, and dynamically adjust within prompts.
• Case Studies: Examples include a customer service chatbot with a low temperature for accuracy and a creative writing assistant with a high temperature for originality.

What is Temperature in Prompt Engineering?

“Temperature” is a parameter used in language models to regulate the randomness of the model’s outputs. Modifying the probability distribution of the model’s predictions modifies the generated text’s level of creativity or determinism.

Lower temperatures sharpen and deterministically enhance the model’s output, preferring high-probability phrases. On the other hand, a higher temperature fosters more inventiveness and unpredictability, making possible a wider range of unpredictable answers.

How Temperature Works?

Temperature is a scalar value applied to the logits (the raw, unnormalized scores output by the model before converting them to probabilities). Mathematically, the probability P(wi) of a word wi​ in the context of the preceding words is calculated as:

zi​ is the logit for the word wi​., and T is the temperature parameter.

When T=1, the logits are unchanged. When T<1, the model’s output distribution sharpens, making high-probability words even more likely. When T>1, the distribution flattens, making the model more likely to sample from lower-probability words.

Practical Implications of Temperature Settings

Here are the practical implications of temperature settings:

1. Low Temperature (0.1 to 0.5)
   • Output Behaviour: The model’s increased concentration and predictability produce coherent text that mostly adheres to the predicted pattern.
   • Use Cases: Perfect for tasks like technical writing, fact-based Q&A, and summarising that demand high precision and dependability.
   • Example: When asked to summarise a piece of writing, a low-temperature setting guarantees that the summary is succinct and closely adheres to the primary ideas of the source material.
2. Medium Temperature (0.6 to 0.8)
   • Output Behaviour: Strikes a balance between coherence and originality, resulting in various responses that are nevertheless pertinent to the question.
   • Use Cases: Ideal for conversational agents, brainstorming sessions, and creative writing where a balance between predictability and creativity is required.
   • Example: A medium temperature enables the model to add new pieces while preserving a logical flow for a creative story challenge.
3. High Temperature (0.9 and above)
   • Output Behaviour: Enhances creativity and randomness, resulting in a less predictable and more varied output from the model.
   • Use Cases: Good for creative tasks that call for a lot of imagination, such writing poetry, fiction, or creative material.
   • Example: when creating poetry, a high temperature might result in original and surprising word and phrase combinations that improve artistic expression.

Also read: Prompt Engineering: Definition, Examples, Tips & More

Best Practices for Using Temperature in Prompt Engineering

Here are crucial practices for using temperature in Prompt Engineering:

• Try Various Temperatures: Begin at a moderate temperature and adjust based on your desired results. Finding the ideal balance between coherence and creativity can be accomplished by adjusting the temperature.
• Context Is Important: Consider the task’s context when choosing the temperature. While a creative writing assignment might fare better with a higher temperature, a technical document might benefit from a lower setting.
• Combine with Other Parameters: The model’s output can be influenced by several parameters, the temperature being only one. The results can be further refined by combining them with other settings, such as top-p (nucleus sampling).
• Dynamic Adjustments: You can improve the outcome of complex activities by dynamically modifying the temperature in different areas of a single prompt. For example, you could set a low temperature for structured sections and a high temperature for creative sections.

Case Studies and Examples

Let’s understand with case studies:

Case Study 1: Assistance for Clients Chatbot

• Goal: Accurately and kindly respond to consumer questions.
• Method: Choose a low temperature (0.3) to ensure the chatbot provides accurate and trustworthy information.
• Result: The chatbot improves customer happiness by providing precise, dependable, factual responses.

Case Study 2: Creative Writing Assistant 

• Goal: Come up with original plot points and story arcs.
• Method: Set the temperature high (0.9) to stimulate the model to generate creative and original content.
• Result: The assistant develops original and surprising plot aspects that inspire writers.

Testing GPT-2 with temperature parameter

import torch
from transformers import GPT2LMHeadModel, GPT2Tokenizer

# Load pre-trained model and tokenizer
model_name = "gpt2"
model = GPT2LMHeadModel.from_pretrained(model_name)
tokenizer = GPT2Tokenizer.from_pretrained(model_name)

The above code will load the GPT-2 model, which can be used for text generation. 

# Function to generate text with a given temperature
def generate_text(prompt, temperature):
   inputs = tokenizer.encode(prompt, return_tensors='pt')
   outputs = model.generate(inputs, max_length=100, do_sample = True,
         temperature=temperature, num_return_sequences=1)
   text = tokenizer.decode(outputs[0], skip_special_tokens=True)
   return text

The above function uses prompt and temperature as arguments and generates text as output. 

# Prompt for generation
prompt = "Once upon a time, in a land far away, there was a"

# Different temperature settings
temperatures = [0.2, 0.5, 0.7, 1.0, 1.5]

# Generate and print text for each temperature
for temp in temperatures:
   print(f"\n--- Temperature: {temp} ---")
   print(generate_text(prompt, temp))

We iterate through different temperatures with the same prompt to observe the differences. 

Below is the output we get

GPT-2 Output to Compare the Temperature

Here’s the full output for better understanding:

Note: These are just outputs from the above screenshots to compare the creativity in different temperatures.

Temperature: 0.2

The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input’s `attention_mask` to obtain reliable results.

Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.

Once upon a time, in a land far away, there was a great war, and the king of the land was slain. And the king of the land was a great king, and he was a great king. The king of the land was a great king, and he was a great king. The king of the land was a great king, and he was a great king. and the king of the land was a great king, and he was a great king. And the

Temperature: 0.5

The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input’s `attention_mask` to obtain reliable results.

Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.

Once upon a time, in a land far away, there was a young man named Rolf. He was a good man, a good man who lived in a village called Kiel. One day he came to a place called the village of Kiel, where he found a man named Rolf. He told him about Rolf’s father, who lived in the village. Rolf told him that Rolf was a good man, and that he had a good mother. Rolf told his

Temperature: 0.7

The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input’s `attention_mask` to obtain reliable results.

Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.

Once upon a time, in a land far away, there was a great battle. The enemy was a great army, and he had gone to destroy them. He was led by the commander of the army, who gave orders that the people of the town of Japheth should be under his command. The commander of the army replied by saying that the people of the town of Japheth would be under his command, and that the people of the town of Japheth should

Temperature: 1.0

The attention mask and the pad token id were not set. As a consequence, you may observe unexpected behavior. Please pass your input’s `attention_mask` to obtain reliable results.

Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.

Once upon a time, in a land far away, there was a prophet of the Jews. He said, “I will not tell anyone what I shall. Yet they say the Prophet of Israel is the Great. They will say I am a prophet. Yet they say he is the prophet.” And it happened that they said, “What do they say? They are talking of his birth; that’s why the prophet is born.” All of that is in the record. It’s a bad record

Temperature: 1.5

Once upon a time, in a land far away, there was a mighty, fierce wind. Then it reached the hills.

When it blew, two, big mountains were coming at them — “two enormous mountains, the tops of which reached the level at the highest place under the earth: and this was a land far away from earth with two huge mountain peaks and some enormous lakes.” Therefore there wasn’t any fire on or in these mountains. The wind and the wind blows would produce a

Also read: Beginners Guide to Expert Prompt Engineering

Analysis of Different Temperature

Here is the analysis of Low, medium, and high temperatures:

• Low Temperature (0.2): The text is predictable and follows a common narrative structure.
• Medium Temperature (0.5-0.7): The text is still coherent but introduces more variety and creativity.
• High Temperature (1.0 and above): The text becomes more imaginative and less predictable, introducing unique and unexpected elements.

Conclusion

Temperature is a potent tool in prompt engineering that allows users to manipulate the originality and predictability of an AI model’s output. By learning and applying temperature settings efficiently, a person can customize the model’s responses to suit certain requirements, be they technical or artistic. Experimentation and careful temperature setting implementation are highly recommended to improve language models’ performance and usefulness in various contexts.

Frequently Asked Questions

Badrinarayan M

Data science Trainee at Analytics Vidhya, specializing in ML, DL and Gen AI. Dedicated to sharing insights through articles on these subjects. Eager to learn and contribute to the field's advancements. Passionate about leveraging data to solve complex problems and drive innovation.