one flow.
one request.
one verdict.
An intrusion detection system that shows TypeSafe's Jev a few flows it has already seen, asks two typed questions about a new one and gets a verdict back, with no text to parse.
Tested on CIC-IDS2017, a public benchmark of network attacks, against a Random Forest, a kNN and an Isolation Forest. The 2,000 test flows are judged in the order they ended, and the true label of each one comes back D flows later, with D from 1 to 200.
-
54%
of zero-day attacks
caught at D = 200 -
29%
for the Random Forest
on the same flows -
≤ 6
false alarms on
1,020 benign flows -
$131
per million flows
at most, list price
With the label one flow late, the Random Forest scores a slightly higher F1, 0.988 against 0.984, a difference the paired test does not confirm. From D = 25 on, each of three runs of Jev IDS scores a higher F1 than every classical detector, and the lead grows with D.
input
flow one CICFlowMeter record from CIC-IDS2017, 83 columns, compact
examples chosen by the Selector, ceil(k) per category
request: two typed questions to jev
is_attack noul Is the flow an intrusion attempt?
category choice Which category does it belong to?
output
is_attack → 0.80
category → DoS, confidence 0.89
verdict → attack p_attack ≥ 0.58, the cut taken before the test
the problem
A flow is the summary of one conversation between two computers. It records how many packets and bytes went each way, how long the conversation lasted, which port it used and dozens of other measures. In CIC-IDS2017 each flow has 83 columns of measures and a label that says whether it is normal traffic or which attack it belongs to.
An intrusion detection system (IDS) receives each flow and decides whether it is benign traffic or part of an attack. When it decides attack, it raises an alarm. Two facts make the task hard.
The first is that attacks appear that the system has never seen. A detector that learned only from older examples tends to take such an attack for normal traffic.
zero-day attack. In network security, an attack the defenders do not yet know when it happens, so they had zero days to prepare. Here, the zero-day attacks are the three attack names of the test that never appear in the system's starting memory. Jev IDS meets them for the first time during the test, with no example of them.
The second is that the right answer arrives late. A security analyst confirms that a flow was an attack only some time later, when the damage shows or someone investigates. Until then the system keeps judging new flows without knowing whether it got the earlier ones right. This delay is called D and is counted in flows. With D = 1 the label of a flow comes back before the next flow is judged. With D = 200 it comes back only after 200 more flows.
how it works
Jev is an AI model from TypeSafe, of a kind called a System One Model (SOM). It reads a state, a block of information, and answers typed questions about it with probabilities, so there is no generated text to parse. Jev IDS puts the flow to judge into the state, next to the instructions, the column names, a description of each category and a few flows whose category is already known, the examples. Then it asks two questions.
-
input
one flow
The record of a single connection, 83 columns, plus the examples the Selector chose.
-
jev
one request
State: the instructions, the categories, the examples and the flow. Questions:
is_attackandcategory. -
output
one verdict
attack · p_attack 0.80
DoS · confidence 0.89
is_attack comes back as a probability from 0 to 1, called
p_attack, and Jev IDS raises the alarm when it is at least 0.58.
category comes back as one of six categories with a
confidence, which says how concentrated Jev's probabilities over the
six are. The values above are a real answer, for the first flow of the
zero-day attack DoS Slowhttptest with D = 200.
the loop
Around that request runs a loop over the stream of flows. The top row of the diagram is one step, and the bottom row is what Jev IDS adds around Jev. The dashed line is the label of each judged flow, which comes back D flows later to the memory and to the KController.
The memory holds the labeled flows a step may draw examples from. These are the flows of the pool, prepared before the test, that ended before the flow under test, and every judged flow whose label has come back.
- The flow arrives. CIC-IDS2017 was recorded by a tool that writes a line when each connection ends, so a real IDS receives a flow only once it has ended. Jev IDS sorts the test flows by the moment each one ended and judges them one at a time in that order, so each step uses only information that already existed.
- The KController decides how many examples. It keeps one number k for each category, and Jev receives k examples of each category, rounded up. Every k starts at 1.
- The Selector decides which examples. Within each category it prefers the flows most similar to the flow under test, the most recent ones and the ones Jev was unsure about, and passes over near-duplicates so the examples show varied cases. Last, it adds the benign flow nearest to the flow under test, so Jev always sees the normal traffic closest to it.
- The compact record shortens the text. Each decimal value reaches Jev with 4 significant digits, so 25160.418 becomes 25160, and every column is kept. Jev charges for the tokens it reads, and this saves 21% to 25% of them with no loss in F1.
- Jev answers and Jev IDS decides. The alarm goes off when p_attack is at least 0.58. That cut was taken before the test, on a separate stream of 300 flows, and the test flows never tuned it.
- The label comes back D flows later. The judged flow joins the memory with its true category. When Jev named the wrong category, the k of the true category doubles, up to 8. When Jev was right, that k falls by 1/k, down to 1 at the least. Every label comes back, missed attacks included.
The KController's rule mirrors the one TCP uses against congestion on the internet (AIMD), which grows the sending window slowly and halves it on a loss. In an IDS the costly mistake is showing too few examples, so the KController grows k fast after an error and shrinks it slowly after a success.
Everything Jev is told lives in one versioned file,
prompts/cicids2017/jev.json. Python adds only the flow and the examples, and the file's sha256
rides with every prediction, so any verdict traces back to the exact
prompt. Examples are labeled by category only, and attack names never
reach a model.
how it was tested
The test uses the relabelled CIC-IDS2017 of Liu et al. (IEEE CNS 2022), public traffic recorded on a lab network over five weekdays in July 2017, with attacks run by researchers. The flows of each label on each day are sorted by time and split in two halves. The earlier half may become examples, and the later half may be judged. The pool, the memory at the start of the test, is a sample of 88,066 flows of the earlier half. The 2,000 test flows were drawn from the later half, 1,020 benign and 980 attacks.
| category | what the traffic is | attack names, with their flows |
|---|---|---|
| BENIGN | normal traffic | 1,020 flows |
| DoS | an attempt to bring a server down with too many requests | DoS Slowloris 101, DoS Slowhttptest 117 (zero-day), DoS Hulk 114, DoS GoldenEye 118 |
| DDoS | the same attack, from many machines at once | DDoS 87 |
| PortScan | a sweep of a machine's ports in search of open services | Portscan 106, Infiltration Portscan 89 (zero-day) |
| BruteForce | guessing passwords by trial and error | FTP-Patator 114, SSH-Patator 87 (zero-day) |
| Botnet | infected machines that obey a remote controller | Botnet 47 |
The three zero-day attack names never appear in the pool, so every detector meets them first during the test. SSH-Patator guesses the passwords of remote logins, DoS Slowhttptest brings a web server down with very slow requests, and Infiltration Portscan is a port sweep run from inside the network by a machine already broken into. Their categories stay in the pool through the other attack names. Together they hold 293 flows, and those flows measure the reaction to an unknown attack.
Each attack name arrives in one burst, a stretch of the stream where its flows come close together. The zero-day bursts span 116 to 171 flows. D = 1, 25, 50 and 100 are shorter than the shortest burst, so some labels of a zero-day attack come back while it still lasts. D = 200 is longer than the longest, so each detector has to recognise the attack without having seen it.
Jev IDS is compared with three classical detectors.
- Random Forest (100 trees) learns from labeled examples of every category. It was the strongest classical detector in this test.
- kNN (4 neighbours) follows the category of the memory flows nearest to the flow under test, in the Selector's own distance, with no call to Jev. It tests whether Jev adds anything beyond finding similar flows.
- Isolation Forest (100 trees) learns only what benign traffic looks like and flags what departs from it. It needs no example of an attack, which makes it the natural rival on zero-day attacks.
The three are refitted at every step on the same memory as Jev IDS, so at each judgement they know exactly what Jev IDS knows. A model trained once before the test would miss the labels that come back during it, and one trained on all the data would see flows from the future. They raise the alarm at p_attack ≥ 0.5, their usual cut.
Jev can answer slightly differently when it gets the same question twice, so Jev IDS ran three times at each D. The classical detectors use fixed seeds and give the same answers every time, so each ran once.
what the numbers mean
Every judged flow falls into one of four cases. An attack with an alarm is a hit. An attack without an alarm is a missed attack. A benign flow with an alarm is a false alarm. A benign flow without an alarm is a correct silence.
- Zero-day recall is the share of the 293 zero-day flows that raised an alarm. Higher is better.
- False alarms count the benign flows, out of 1,020, that raised an alarm. Lower is better, because each one costs an analyst time.
- F1 joins the two goals, catching attacks and avoiding false alarms, in one number from 0 to 1 that is high only when both are good. It is 2 × hits divided by (2 × hits + false alarms + missed attacks).
- The paired test looks only at the flows where two detectors disagree. If both were equally good, those flows would split half and half, like a coin. The exact sign test gives p, the chance of a split at least this uneven, and below 0.05 the difference counts as real.
evidence
The paper-v2 split holds 2,000 flows of CIC-IDS2017 in
the order they ended, drawn from rows no earlier split held and read
once, in the runs of 2026-11-02. Jev IDS is the mean of its three runs
and its false alarms the range over them. Each classical detector ran
once.
| D | f1 | zero-day recall | false alarms / 1,020 | |||||
|---|---|---|---|---|---|---|---|---|
| jev ids | rf | knn | if | jev ids | rf | jev ids | rf | |
| 1 | 0.984 | 0.988 | 0.983 | 0.495 | 0.953 | 0.945 | 1 to 2 | 1 |
| 25 | 0.975 | 0.961 | 0.951 | 0.494 | 0.883 | 0.799 | 2 | 0 |
| 50 | 0.969 | 0.947 | 0.934 | 0.494 | 0.861 | 0.713 | 3 to 6 | 0 |
| 100 | 0.954 | 0.906 | 0.898 | 0.495 | 0.721 | 0.471 | 2 to 4 | 0 |
| 200 | 0.923 | 0.873 | 0.869 | 0.495 | 0.540 | 0.294 | 3 | 0 |
With D = 1, Jev IDS and the Random Forest are level, because both learn almost at once from labels that come back one flow later. As D grows, the Random Forest falls fast, above all on zero-day attacks. Jev IDS falls more slowly, because Jev can judge an attack it has never seen from examples of the known ones. The Random Forest raises almost no false alarm, so the flows it loses are attacks it lets through. At D = 200 it misses all 87 flows of SSH-Patator and all 89 of Infiltration Portscan, where the first run of Jev IDS catches 62 and 28. The Isolation Forest never uses the returning labels, finds about 35% of all attacks at every D and raises 75 to 77 false alarms.
Panel (a) shows F1 and panel (b) zero-day recall, both against the delay D, and higher is better in both. The delays are evenly spaced, so the step from 100 to 200 takes the same width as the step from 1 to 25.
Flow by flow, Jev IDS beats the Random Forest on zero-day attacks in every run from D = 25 on (adjusted p ≤ 0.042) and on all flows in every run from D = 50 on (adjusted p ≤ 0.0014). At D = 25 the win on all flows holds in two runs of three. At D = 1 no run differs from the forest on all flows, and in one run the forest is better on known attacks. Longer delays add nothing past D = 200, because the gap stops growing once the delay outlasts the bursts.
what each part adds
Each part of Jev IDS was removed or swapped in turn, one run each at D = 1, 50 and 200, and compared with the full system.
| line | what changes | f1 | false alarms |
|---|---|---|---|
| full system | nothing | 0.985 / 0.970 / 0.925 | 2 / 5 / 3 |
| no selector | examples drawn at random | 0.925 / 0.935 / 0.886 | 69 / 65 / 58 |
| fixed k = 1 | no KController | 0.978 / 0.962 / 0.916 | 3 / 3 / 4 |
| fixed k = 4 | no KController, 4 per category | 0.990 / 0.978 / 0.928 | 6 / 6 / 8 |
| full record | every decimal kept | 0.982 / 0.972 / 0.923 | 2 / 3 / 3 |
| random, k = 4 | no Selector, no KController | 0.957 / 0.955 / 0.927 | 75 / 73 / 69 |
| no nearest benign | no benign flow added | 0.979 / 0.971 / 0.929 | 27 / 29 / 22 |
The Selector matters most. Without it, Jev IDS loses on all flows at every D and raises 58 to 69 false alarms. The KController buys 0.7 to 0.8 points of F1 over one example per category for 5% to 10% more tokens, and four examples per category tie with the full system for twice the tokens. The nearest benign flow cuts the false alarms from 22 to 29 down to 2 to 5. The kNN, which reads the nearest flows without Jev, ties with Jev IDS at D = 1 and loses from D = 25 on, in the paired test of the first run.
limits worth knowing
- One dataset carries the stream results, and the delays were chosen from the bursts of this same split.
- Jev IDS ran three times at each D and every ablation once. Its F1 varies by up to 0.009 between runs, so a smaller gap between two ablation lines may come from the run alone.
- The cut of 0.58 sits on a thin margin. On the validation stream, the highest benign p_attack was 0.55 and the lowest attack p_attack 0.58.
- Every label comes back, missed attacks included. A real security team confirms only some of them.
- Cost is tokens times TypeSafe's list price of $0.042 per million input tokens, with output free, and the bill may differ. Latency is the wall clock of one successful call, a median of 0.22 to 0.24 s per flow.
earlier benchmark: nsl-kdd
Before the stream, Jev IDS was measured on NSL-KDD as a k-shot detector, with examples drawn at random and no feedback, against Gemini 3.6 Flash, a Random Forest and an Isolation Forest. At k = 1 it was 5.4 times faster and 22 times cheaper than the LLM, as precise (0.953 against 0.942) and lower on F1 (0.856 against 0.880). The tables are in docs/results.md.
try it
git clone https://github.com/jev-sec/jev-ids.git
cd jev-ids
uv sync
.env holds TYPESAFE_API_KEY for Jev.
Download CICIDS2017_improved.zip from the
relabelled version of Liu et al.
into data/raw/cicids2017/ and prepare it once.
uv run python -m scripts.prepare_cicids2017
uv run jev-ids run --dataset data/cicids2017/dataset.json --detector jev --split smoke --delay 1 --k 1 --control aimd --selection score+nearest --seeds 0
The smoke split is five flows. The run writes
results/<timestamp>-cicids2017-jev-smoke/ with
config.json, one JSON row per flow in
predictions.jsonl and the raw answers in
responses.jsonl. The README shows how to run the
classical detectors and reproduce every number on this page.
bring your own flows
Jev IDS is an independent research prototype. Its numbers come from two benchmarks, CIC-IDS2017 and NSL-KDD. It can still sit inside a commercial solution, and this is how.
where jev fits
A commercial IDS already has sensors, a flow exporter and a signature engine feeding a SIEM. Jev IDS sits beside that pipeline, off the packet path, and judges one flow record at a time.
- A net behind the signatures. Signatures miss what they have never seen. Sample the flows the engine passed as clean, or every flow to a critical asset, and let Jev judge them. On CIC-IDS2017, with labels 200 flows late, it caught 54% of the zero-day attacks where a Random Forest caught 29%.
- Learning from the analysts. Every alert an analyst confirms or dismisses goes back into the memory as a labeled flow, and the Selector and the KController use it from the next request on, with no retraining.
-
A category for the playbook. The
choiceanswer names the category with a confidence, so the SIEM routes DoS, PortScan and BruteForce, or your own taxonomy, to different runbooks with no parser in between. - Coverage from day one. A new site or tenant has no training set. Jev starts from one labeled flow per category, so it covers the segment while a classical model is still collecting data.
A quarter of a second per verdict and a rate-limited API make this an
asynchronous side channel, fed from the exporter (NetFlow, IPFIX, Zeek
conn.log) through a queue, never an inline filter.
six steps
-
Get a key. Jev is served by TypeSafe. Set
TYPESAFE_API_KEYin.env. Pricing and terms are TypeSafe's. -
Describe your flows. Write a dataset card like
data/cicids2017/dataset.json, with the columns your flow exporter emits in order, which of them are symbolic, your categories and which one is benign. -
Write the request. Copy
prompts/cicids2017/jev.json, replacecolumnsand the category descriptions with yours, and keep the two questions. - Seed the memory. One labeled flow per category from your own network is enough to start, because the KController begins at k = 1 and raises k only for the categories Jev gets wrong.
-
Call the detector.
JevDetector(load_prompt(path)).predict(flow, examples)returnsp_attack,category_predandconfidencefor one flow in about a quarter of a second, andstream.pyshows how the memory, the Selector and the KController chooseexamplesat each step. Routep_attackto your alerting with the cut your alarm budget allows. 0.58 was the benchmark's choice, not a rule. -
Measure before you trust. Run
jev-ids run --delayandjev-ids metricson a labeled split of your own flows. The numbers above are CIC-IDS2017's, not yours.
Only flow features leave your network, never payloads, but they do leave it. Every request goes to TypeSafe's API.