I don’t trust feelings. I trust logs.
I have recently worked on several projects that made me question my basic instincts when it comes to using different AI models in day-to-day work. One of them was the thought of what happens after you type a sentence (prompt), and the AI's backend starts typing.
I manually measured the time of these replies and found myself saying that this feels faster. However, feelings can be misleading, so I built a small setup that measures what is happening and somehow provides proof of the results, which anyone can try and verify.
In this post I walk the full path starting from a prompt that I sent to an answer from AI, where it shows how an LLM turns text into tokens (integers called token IDs), how it chooses the next token, how this process is measured when it comes to speed with time to first token (TTFT) and tokens per second (TPS), the tests I do about code generation with pass@1 / pass@k, and the proof of what happens is based on a hash-chained receipt and a verifier.
The entire process can be completed in less than two minutes.
Part 1: From your words to the model answer
Your words to the model’s response
Since models do not read from our language, but rather, they read tokens, the word “recursion” would be broken down into “recur” and “sion” tokens. In this case, the tokenizer maps text to tokens and back by using algorithms such as Byte Pair Encoding.
The tokenizer is a separate component from the model, and its features are important in practice. Let's say that a single trailing space can change the IDs, for instance, changes like “Egg” versus “egg” may map differently, while numbers and punctuation can split in ways that are not intuitive. Therefore, languages other than English often expand to more tokens for the same idea, which affects both speed and cost. A larger vocabulary packs more text into fewer tokens, so for a fixed token limit, you effectively fit more content.
How the model picks the next token
From every prompt that we share when talking to an LLM, the model predicts a distribution over the next token, and we choose how to pick from it.
Greedy search - refers to a helper that doesn't think ahead or try new approaches. Instead, it selects the best choice for now, which is the most likely next word, potentially leading to drifting into loops or a very mechanical feel.
Temperature - It's a switch that dictates the curve. The lower values keep the model close to the safety and predictability, while the higher values flatten this curve and allow less predictable responses. When the temperature approaches zero(T \to 0), the output becomes the same as the greedy choice.
Top-p (nucleus) sampling - Selects from the smallest set of tokens whose combined probability is at least p (0<p\le1), setting p<1 trims the low-probability tail and often reduces repetition while keeping variety; p=1 disables the nucleus filter.
Top-k sampling - keeps only k most likely tokens, and many systems mix p<1 and top-k to adjust how creative or repetitive the output is.
Beam search - Unlike greedy search, where you always grab the single best next block and keep going, beam search doesn't limit you to just one path. Instead, you keep a few of your best mini-stories at the same time to generate structured outputs.
Streaming back to you
When it comes to streaming tokens back to the user, the model does this as they are generated, and you see the answer arrive one piece at a time. This process typically utilizes Server-Sent Events. This is sufficient to quantify TTFT, indicating responsiveness, and TPS for throughput after the first character is received.
An example run from my terminal:
TTFT: 1205 ms | ≈tokens/sec: 57.25
Saved: results/eval-1756759274290.json
Attestation OK (16 records)Part 2: Measure, evaluate, and prove
Measure
During this experiment, logging the stream is one of the first steps to calculate TTFT and the approximate TPS in the stream. The reason TTFT can jump around is due to internet latency, server queues, or a cold start, which can delay the first byte. However, TPS remains steady because once the stream is flowing, we're just counting tokens over time, and the math smooths out short hiccups.
Evaluate
I created a set of 16 JavaScript tasks with unit tests. For each task, I asked the model to write exactly one function, which would run in a sandbox. To get a fair baseline, I use greedy decoding (deterministic baseline: T=0, p=1, do_sample=false). If the API allows me to set a seed, I do so, so that rerunning the same test gives the same output. For more info on a formal definition used in code-generation evals, here's the metric section: HumanEval here.
Proof
Each task's result is like a receipt and is stored in a JSON file. Each line stores its own hash and the previous line's hash. A verifier recalculates every hash and checks the link. If every link matches, it prints "Attestation OK"; if anything was changed, it fails. We also store a hash of the prompt in a separate manifest, which proves not just the output but what was asked. So in other words, you're not asking people to "trust me", but you're giving them a proof trail that will clearly show if anyone changed it after the fact.
Results with deterministic settings
Settings
T=0
p=1
k=1
Suite - 16 small JavaScript tasks with unit tests (IDs: two-sum, merge-intervals, roman-to-int, …)
Tokenizer - exact counts, so TPS is meaningful
ReadiModel pass@1 pass@k Median latency Avg tokens (in/out)
GPT-4o-mini 16/16 16/16 ~3145 ms ~25 / ~75
GPT-4o 16/16 16/16 ~1573 ms ~25 / ~79ngGPT-4o-mini and GPT-4o both solved every task on the first try, which is what pass@1 means on this suite. In my runs, GPT-4o had a lower median task latency than GPT-4o-mini. Input token counts were similar. Output tokens were slightly higher for GPT-4o, which matches the more explicit code it tended to write. Latency varied somewhat from run to run, which is normal with a network in the loop, so I rely on medians and publish the raw files.
Code you can run
Streaming to measure time to first token and tokens per second
ts
// scripts/stream.ts (mini)
const t0 = Date.now();
const r = await fetch(`${BASE_URL}/chat/completions`, {
method: "POST",
headers,
body: JSON.stringify({
model: MODEL, stream: true, temperature, top_p,
messages: [{ role: "user", content: prompt }],
}),
});
if (!r.ok || !r.body) throw new Error(await r.text());
const dec = new TextDecoder();
let buf = "", first = 0, chars = 0;
for await (const chunk of r.body as any as AsyncIterable<Uint8Array>) {
buf += dec.decode(chunk, { stream: true });
let i;
while ((i = buf.indexOf("\n\n")) !== -1) {
const evt = buf.slice(0, i).trim(); buf = buf.slice(i + 2);
const line = evt.split("\n").find(l => l.startsWith("data: "));
if (!line) continue;
const data = line.slice(6).trim();
if (data === "[DONE]") break;
const delta = JSON.parse(data).choices?.[0]?.delta?.content ?? "";
if (delta) { if (!first) first = Date.now() - t0; chars += delta.length; }
}
}
const total = Date.now() - t0;
const toks = Math.max(1, Math.round(chars / 4));
const tps = first ? +(toks / ((total - first) / 1000)).toFixed(2) : 0;
console.log(`TTFT: ${first} ms | ≈tokens/sec: ${tps}`);Verifier to check the receipts
ts
// src/verify.ts (mini)
import fs from "fs"; import crypto from "crypto";
const sha = (s:string)=>crypto.createHash("sha256").update(s).digest("hex");
const stable = (o:any)=>JSON.stringify(Object.keys(o).sort().reduce((a,k)=> (a[k]=o[k],a), {}));
const file = process.argv[2];
const lines = fs.readFileSync(file, "utf8").trim().split("\n").filter(Boolean);
let prev:string|null = null;
for (const ln of lines) {
const { hash, idx, prevHash, ...rest } = JSON.parse(ln);
const expect = sha(stable({ ...rest, prevHash: prev }));
if (prevHash !== prev || hash !== expect) throw new Error(`Chain mismatch at idx ${idx}`);
prev = hash;
}
console.log(`Attestation OK (${lines.length} records)`);Commands to reproduce
npm i
npm run stream -- "Explain recursion in one paragraph with a short JS example."
K_ATTEMPTS=1 npm run eval
npm run verify -- $(ls -t results/attest-*.jsonl | head -n1)
npm run summarize -- $(ls -t results/eval-*.json | head -n1)Full streaming script and attestation writer:
https://github.com/kju4q/prompt-to-proof.git
What I learned while building this
I'm sure that each of us naturally judges the speed of how an LLM responds by how fast the answer shows up, but I think that's vague, because now I split it into:
Time to First Token (TTFT)
Tokens per second
Knowing that the model reads tokens, even minor changes, such as adding a space at the end or writing in another language, can significantly alter the token count, leading to higher costs and sometimes more delay.
When T=0 and p=1, the model becomes repeatable, which lets me ask a clean question: did it pass on the first try? That is pass@1.
When I turn sampling back on and allow a few tries, I use pass@k which shows whether the model can find a solution if its allowed to explore different outputs, some problems flip from fail to pass once the model gets a few shots.
Saying "16/16 passes" is just a claim, but writing results to a hash-linked receipt file and verifying it is proof. If anyone edits a line, the chain breaks and the verifier complains, which turns screenshots into evidence people can check in seconds.
Mindset shift
I stopped debating from memory and started comparing the artifacts. When the tasks, prompts, and settings are fixed, the comparisons get calm and fair, because reading a scoreboard is more truthful than arguing opinions.