Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Tokens are not multiple words, but are actually usually parts of words. The rule of thumb OpenAI uses is that there will be 100 tokens for every 75 words.

If you want to just see the tokens for yourself, though, just enter some text here:

https://platform.openai.com/tokenizer



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: