Language Model API Parameters

PublishedMay 31, 2023

•2 min read•View as Markdown

Mark Vasilkov

Comments

Join the discussion

No comments yet. Be the first to comment.

More from this blog

The making of King Thirteen

Hey hey people, I wrote a game recently for a certain coding competition, and thought I'd share the thought process behind it. Let's start with the Design King Thirteen is by far the most deliberately designed game I've made. It all began with a very...

Sep 21, 20247 min read2.7K

Super Castle Game (js13kGames–2023)

An open mind is like a fortress with its gates unbarred and unguarded. This is the story of the Super Castle Game, a js13k game jam entry. Naturally, it begins with the theme of the 2023 edition of the compo: 13th Century. When it dropped, I was com...

Sep 16, 20238 min read348

CSV field larger than field limit in Python

So I was working on something very simple involving a CSV file: import csv with open('file.csv', 'r') as f: reader = csv.reader(f) lines = [ln for ln in reader] And as one would expect, the result of running this code was Error: field large...

Apr 24, 20231 min read1.8K

CSV field larger than field limit in Python

ZX Spectrum system font

Old computer fonts are fascinating. Making a consistent-looking font is a remarkable achievement in its own right, but designing a monospaced font that is readable on a very low-resolution screen, is aesthetically pleasing, and has a distinct persona...

Feb 13, 20232 min read1.3K

Mark Vasilkov

12 posts

Temperature

The temperature (a floating-point number in the range 0.0–1.0) is used for sampling during response generation, which occurs when topP and topK are applied. Temperature controls the degree of randomness in token selection. Lower temperatures are good for prompts that require a more deterministic and less open-ended or creative response, while higher temperatures can lead to more diverse or creative results. A temperature of 0 is deterministic, meaning that the highest probability response is always selected.

Top-K

Top-K (an integer in the range 1–40) changes how the model selects tokens for output. A top-K of 1 means the next selected token is the most probable among all tokens in the model's vocabulary (also called greedy decoding), while a top-K of 3 means that the next token is selected from among the three most probable tokens by using temperature.

For each token selection step, the top-K tokens with the highest probabilities are sampled. Then tokens are further filtered based on top-P with the final token selected using temperature sampling.

Specify a lower value for more deterministic responses and a higher value for more diverse responses. The default top-K is 40.

Top-P

Top-P (a floating-point number in the range 0.0–1.0) changes how the model selects tokens for output. Tokens are selected from the most (see top-K) to least probable until the sum of their probabilities equals the top-P value. For example, if tokens A, B, and C have a probability of 0.3, 0.2, and 0.1 and the top-P value is 0.5, then the model will select either A or B as the next token by using temperature and excludes C as a candidate.

Specify a lower value for more deterministic responses and a higher value for more diverse responses. The default top-P is 0.95.

Command Palette

Comments

More from this blog

Temperature

Maximum Output Tokens

Top-K

Top-P