Atlan Team
Introduction
In 2020 we released SharpML, a proof of concept that used Machine Learning to help identify potential passwords in accessible file shares. The original idea was simple: internal file shares contain huge amounts of information, most of it uninteresting, and we wanted to see whether a model could help an operator find the useful parts without relying entirely on grepping for the word password.
The original version was built for offensive security. It collected data from a file share, scored candidate passwords and, where a username and password pair was identified, could validate the combination against Active Directory.
We have changed how we release research since then. We no longer release offensive tooling publicly. For this experiment we wanted to revisit the useful part of SharpML - finding potentially sensitive material in very large amounts of ordinary data - and turn it into something that can be used defensively by a SOC.
That became SharpMLv2: a read-only secret exposure triage tool. It scans file shares, detects candidate secrets, removes the raw secret from the record, and produces structured findings for analyst review. It does not attempt to authenticate credentials, spray passwords or validate accounts.
At the same time, TypeSafe AI released Jev, a model designed to return typed decisions rather than free-form text. Jev's public documentation defines Noul as the probability of yes, and its developer material describes its outputs as calibrated probabilities. That gave us a useful question to test:
When Jev returns 0.87 for a security decision, does 0.87 behave like an 87% probability on the task we actually care about, or is it simply a strong model score?
This post documents the tool, the SMB test, the calibration harness and the first measured results. It also documents a flaw we found in our own synthetic corpus generator. That flaw is important, because a calibration experiment is only as good as the ground truth used to assess it, so it is not ideal.
From SharpML to SharpMLv2
The old SharpML workflow collected raw file-share data and tried to identify likely passwords. The model assigned a likelihood score based on characteristics learned during training, and the C# wrapper could then perform an Active Directory authentication check.
SharpMLv2 changes both the objective and the outcome. We are not trying to prove that a credential works. We are trying to help a defensive team answer a different question: which of the thousands of secret-looking strings in an environment are worth investigating?
The important point is that Jev is not being asked to magically discover whether an arbitrary hash or opaque identifier is malicious. SharpMLv2 gives it contextual evidence: the detector that fired, the key name, file extension, redacted surrounding lines, value length, entropy, whether the value resembles a placeholder, and selected path hints. The raw secret is not sent to Jev.
Why Jev?
For this use case we are interested in classification rather than generation. We do not need a model to write an explanation for every candidate secret. We need structured answers that software can consume.
Jev exposes three question types that fit this well: Choice, Score and Noul. Choice selects from a bounded set, Score returns a probability-weighted position across ordered levels, and Noul returns the probability of yes. Jev's documentation also makes an important distinction: typed output guarantees the shape of the answer, not factual correctness.
SharpMLv2 therefore asks questions such as:
{
"secret_type": {
"type": "choice",
"instructions": "What kind of sensitive authentication material does this record most likely represent?"
},
"likely_real": {
"type": "noul",
"instructions": "Is this likely to be a real deployed secret rather than example, documentation, placeholder or test data?"
},
"priority": {
"type": "score",
"instructions": "How urgently should a defensive security analyst review this potential secret exposure?"
}
}For the direct API test we used the TypeSafe endpoint https://api.typesafe.ai/v1/systemone with the model identifier jev-latest. The model identifier is recorded here because the public Playground currently displays a different identifier, which will error in the code if run using the Playground model version.
Probability is not just a bigger confidence number
This is the part we wanted to test properly, because it is easy to look at a number such as 0.87 and read more into it than the model has actually demonstrated.
Suppose SharpMLv2 supplies all of the relevant evidence it has for a candidate and asks Jev whether the record is likely to be a real deployed secret. Jev returns 0.87.
Those two meanings are actually pretty different and ultimately important in the context of the security work we all do. Defining the question and the possible answers tells the model what yes and no mean. It does not, by itself, prove that the numerical output is calibrated against the population in a particular SOC environment.
Jev's public material describes its probabilities as calibrated. For this experiment we want to have a look at that, and to see how useful it can be for both offensive and defensive security work where ultimately calibration is important.
What we mean by calibration
Calibration is easiest to understand by ignoring the maths for a moment. Imagine a SOC model marks 100 findings as 0.80. If the model is well calibrated, somewhere around 80 of those findings should turn out to be real positives and around 20 should not.
If only 40 of the 100 are actually positive, then 0.80 is over-confident for that population. If 98 are positive, the model is under-confident. A useful probabilistic model should not just put the right cases near the top; the numbers themselves should mean something.
Brier score explanation without deep statistics
The Brier score gives the model a penalty for every probability it produces. For a binary question, the real outcome is written as 1 when the answer was yes and 0 when the answer was no. We subtract the real outcome from the model probability, square the difference, and then average those penalties across all cases.
For a binary outcome, a Brier score falls between 0 and 1. A score of 0 would mean every probability prediction was perfect. A score of 1 would mean the model was maximally and confidently wrong on every case. As a simple reference point, predicting 0.50 for every case always produces a per-case Brier penalty of 0.25.
Expected Calibration Error (ECE), explained simply
ECE looks at the problem differently. Instead of scoring every prediction individually, it places predictions into buckets - for example 0.0-0.1, 0.1-0.2, all the way up to 0.9-1.0.
For each bucket we compare two things:
- the model's average predicted probability in that bucket; and
- the fraction of cases that were actually positive.
If the model's 0.8-0.9 bucket averages 0.85 but only 60% of those findings are actually positive, that bucket has a calibration gap of 0.25. ECE takes those gaps across all buckets and weights them by how many predictions landed in each bucket.
ECE is also not magic. Its value changes depending on how the bins are chosen, and a single number can hide where the model is over-confident or under-confident. For that reason, a proper calibration assessment should also inspect the reliability bins or reliability diagram rather than treating ECE as a standalone verdict.
Most importantly, neither Brier nor ECE means much unless we know which cases were actually evaluated and whether those cases represent the population we intended to test. That detail became particularly important in this experiment.
Experiment 1 - The SMB needle in the haystack
The first test was deliberately simple. Before looking at Jev, we wanted to prove that SharpMLv2 could traverse an SMB share, process a large amount of ordinary data, detect a planted secret and preserve the redaction boundary.
The Windows lab share contained 15,302 files in total. The corpus included 15,000 ordinary files, 300 obvious placeholder-secret decoys, a manifest and one synthetic production-looking needle. SharpMLv2 was pointed at the UNC path rather than the underlying local directory.
| SMB test measurement | Result |
|---|---|
| Total files in share | 15,302 |
| Candidates examined | 422 |
| Duplicates skipped | 121 |
| Findings reported | 301 |
| Warnings | 0 |
| Planted needle found through UNC path | Yes |
| Raw planted secret present in JSONL output | No |
The planted finding was identified as a password in a production-looking finance/database path, with a local p(real) score of 0.75. The report retained the test marker and surrounding context but replaced the actual value with a redacted length marker.
That gives us evidence that the defensive scanner itself works across SMB and that the data-handling boundary behaves as intended. It does not say anything about Jev calibration yet.
Experiment 2 - Jev probability calibration
The calibration harness was intended to create 5,000 ordinary noise files plus 1,000 labelled candidate cases. The first corpus was configured for 200 positives and 800 negatives, and a second corpus was configured for 50 positives and 950 negatives to explore a lower positive base rate.
SharpMLv2 then ran in --jev-only mode so that the probability being evaluated was the raw Jev Noul output rather than a blend of Jev and local heuristics.
What the first run appeared to show
On the first surviving set of 300 labelled cases, the local heuristic baseline produced a Brier score of 0.095900 and ECE of 0.283333. Jev produced a Brier score of 0.033699 and ECE of 0.160500 on the same 300 cases.
| 300-case surviving subset | Brier score | ECE |
|---|---|---|
| Local heuristic | 0.095900 | 0.283333 |
| Jev | 0.033699 | 0.160500 |
On the second surviving set of 150 labelled cases, Jev produced a Brier score of 0.045599 and ECE of 0.172867.
Qualitatively, the output also looked sensible. Production-style API keys and passwords were generally scored in the 0.8-0.9 range, obvious example values around 0.04-0.05, test fixtures around 0.13-0.16, and the deliberately difficult production-path migration fixtures around the 0.4 range.
What the calls cost
Cost is part of whether a triage design is actually deployable, so the API usage behind these two runs is worth stating alongside the metrics. Both runs fell between 18 and 19 September 2026.
Then we found a problem in our own test harness
The intended corpus sizes and the actual file counts did not agree. A 5,000-noise plus 1,000-labelled corpus should have produced 6,002 files once the ground-truth CSV and manifest were included. The first share contained 5,302 files and the second contained 5,152.
Reviewing the corpus generator showed why. The negative-case directory names reused $i % 100. Because each category advanced in steps of four, multiple labelled negative cases were written to the same paths and later cases overwrote earlier ones. The ground-truth CSV still contained 800 or 950 negative rows, but only 100 negative files survived on disk.
That is exactly why the evaluator matched 300 / 1000 cases in the first corpus and 150 / 1000 in the second. These were not clean 20% and 5% prevalence experiments. The effective surviving sets were 200 positive + 100 negative, and 50 positive + 100 negative respectively.
We therefore do not present the numbers above as a final validation of Jev's calibration claim. They are useful pilot measurements on the cases that survived the generator collision, and they show Jev outperforming our simple local heuristic on that subset, but they do not test the population we originally intended to create.
This is also a useful reminder about model evaluation in general. It is easy to calculate a Brier score and ECE and get numbers that look scientific. The difficult part is making sure the population behind those numbers is actually the population you think you tested.
What SharpMLv2 sends to Jev
One of the main design decisions was not to send discovered secret values to an external model. The candidate is converted into a redacted record before classification.
{
"extension": ".ini",
"detector": "password_assignment",
"key_name": "password",
"masked_value": "<REDACTED:length=42>",
"value_length": 42,
"entropy": 4.402,
"looks_placeholder": false,
"path_hints": ["finance", "production", "config"],
"file_name_hints": ["database"],
"surrounding_context": [
"host=prod-db-lab.invalid",
"username=svc_demo",
"password=<REDACTED>"
]
}This does not make the record non-sensitive. File names, path hints and surrounding configuration can still reveal information about an environment, so production use requires an explicit data-handling decision. The lab used synthetic data only.
What we can conclude
The SMB part of the experiment is straightforward: SharpMLv2 works successfully under .NET 8, scanned a real UNC path, found the planted secret in a 15,302-file haystack, redacted the value, and completed without scanner warnings.
The Jev integration also worked once the direct API was configured with the model identifier accepted by the endpoint. The model produced structured Choice, Score and Noul outputs that were operationally useful for triage.
On the surviving 300-case pilot subset, Jev's raw Noul outputs had materially lower Brier and ECE values than our deliberately simple local heuristic. That is promising evidence that the model's probabilities carry more useful information than the heuristic score in this particular synthetic task.
What we cannot claim from this run is that Jev is calibrated at the intended 20% or 5% population base rates. The corpus-generation collision changed the evaluation population before the model saw it. A corrected generator and rerun are required before making that more empirical claim.
We also do not treat a high Jev score as proof that a credential is valid. SharpMLv2 deliberately does not authenticate discovered credentials.
Limitations
A synthetic lab is not a corporate file share. Naming conventions, secret prevalence, duplication, access controls and file types will differ in real environments.
Calibration belongs to a particular task, model version and input distribution. Even a clean result on a corrected SharpMLv2 corpus would not mean that a Jev score of 0.87 should be read as 87% for every unrelated security decision.
SharpMLv2's deterministic candidate detector also controls what Jev gets to see. If the local detector never identifies a secret-like record, Jev cannot recover it. End-to-end secret discovery therefore needs to be evaluated separately from model calibration.
Release Position
The original SharpML was offensive research. Our public release position is different now and we do not intend to publish tooling whose purpose is credential validation or account access.
SharpMLv2 is therefore designed as a SOC tool: read-only discovery, secret redaction, bounded classification and analyst-facing findings. The aim is to reduce the amount of low-value material a defender has to review, while keeping the final security decision with the operator.
The defensive research harness can be released without the Active Directory validation behaviour of the original proof of concept. That lets us continue publishing useful research while keeping offensive capability out of the public tool. The code will be published at github.com/AtlanDigitalR-D/SharpMLv2.
Conclusion
Our original SharpML work asked whether Machine Learning could help find a useful password candidate in a large amount of file-share data. SharpMLv2 asks a more defensive question: can a fast classification model help a SOC distinguish genuine exposure from examples, placeholders and test values?
The answer from the first part of the experiment is yes: the scanner can find a planted secret across SMB and preserve the redaction boundary. The first Jev run is also encouraging: on the cases that actually survived into the corpus, its probability outputs were substantially better than our simple heuristic baseline which is good news for JevAi.
But the frustrating, while important result is the error in the corpus generation. Our first calibration corpus was wrong. The metrics were calculated correctly for the surviving files, but the generator had silently overwritten cases and changed the test population. We are therefore treating those calibration numbers as interim evidence rather than a verdict, and we will come back and fix it later.
References