另外如果你有“送出 token 數會超過 4096 這個限制” 問題困擾,想要先算有幾個 token used 在送出 API call 前的話,可以參考這個 notebook :
https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb
我自己有用也有確認過,和 API 回回來的 token 數量是一樣的
sample code 大概會長這樣:
``` for message in messages:
num_tokens += 4 # every message follows <im_start>{role/name}\n{content}<im_end>\n
for key, value in message.items():
num_tokens += count_token(value, model_engine)
if key == "name": # if there's a name, the role is omitted
num_tokens += -1 # role is always required and always 1 token
num_tokens += 2 # every reply is primed with <im_start>assistant```
```import tiktoken
def count_token(text, model_engine):
# ref:
https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb
try:
encoding = tiktoken.encoding_for_model(model_engine)
except KeyError as err:
print(err)
print(f"Use default encoding model: cl100k_base")
encoding = tiktoken.get_encoding("cl100k_base")
num_tokens = len(encoding.encode(text))
return num_tokens```