Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I have a latency sensitive application - anyone know if any tools that let you compare time to first token and total latency for a bunch of models at once given a prompt. Ideally, run close to the DCs that serve the various models so we can take out network latency from the benchmark.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: