hpcfg - High-Performance PCFG trainer

A PCFG trainer. It trains, and hashcat guesses.

Official repository: https://github.com/matrix/hpcfg

This document describes v2.0.

Its default output is a ruleset hashcat reads, in the same 4.x layout pcfg_cracker, pcfg-go and Cpcfg use. Handing it to pcfg_cracker or pcfg-go takes --ruleset-format pcfg-cracker: the default declares hpcfg's own version, which pcfg_cracker refuses, and leaves out the lists its loader opens regardless. Cpcfg wants neither of those, and takes --ruleset-format cpcfg instead - it reads two files of the escape in a shape of its own.

make
./hpcfg -t rockyou.txt -g /tmp/ruleset
./hpcfg -t hibp.txt -g /tmp/ruleset -w        # lines are count:password
hashcat -m 0 -a 4 hashes.txt /tmp/ruleset

Building

A C compiler and pthreads, nothing else: no configure and no dependencies to fetch. Go and Python are needed only to regenerate the tables under src/ and to run the tests. make picks the OS up from uname.

Linux, Android	make
macOS	make
FreeBSD, OpenBSD, NetBSD, DragonFly	gmake
Windows, under MSYS2 or Cygwin	make, which writes hpcfg.exe
Windows, cross-built from Linux	make CC=x86_64-w64-mingw32-gcc

Any other uname is refused by name rather than failing later in the compile. LTO is on where it works and off where it is known not to. ENABLE_LTO=0 turns it off by hand.

make info      # the OS it detected, the compiler target, whether LTO is on
make native    # -march=native: faster, and runs only on a CPU like this one
make clean

flag
-t, --train <file>	training wordlist, - for stdin
-g, --ruleset-save <dir>	ruleset to write, on a path that does not exist yet: a run adds files to the directory it is given, so it refuses one that is already there rather than mixing two rulesets
-w, --weighted	weighted input, <count><sep><password>
-s, --sep <chars>	bytes that may separate the count (default :)
-f, --terminal-count-min <n>	drop terminals seen fewer than n times, decided when the lists are written, so the count is exact and n has no ceiling
-m, --mw-file <file>	words to know before the corpus is read
--count-min <n>	with --weighted, skip lines whose count is below n
-c, --coverage <0..1>	the grammar's share against OMEN's, where 0 is OMEN alone and 1 turns it off
-S, --sensitive-enable	also write the addresses and the URLs themselves
-e, --encoding <name>	what to tell a reader the values are in. It labels them, it does not convert them
-C, --comments <text>	a note to carry in config.ini
--ruleset-format <name>	who the ruleset is for: hashcat (default) writes what hashcat reads and what a merge reads back, pcfg-cracker adds what pcfg_cracker and pcfg-go need to load it, cpcfg writes the escape's two entry files the way Cpcfg reads them
--rulesets-merge <dir>	fold rulesets together instead of training, with --rulesets-merge-weights a:b to set the shares
-i, --ruleset-info <dir>	what the ruleset holds, and what is wrong with it
--ruleset-diff <dir>	with -i, how much of the mass two rulesets agree on
--mw-threshold, --mw-len-min, --mw-len-max, --multiword-disable	the splitting policy, which decides the terminals. The two lengths are 1 to 128, which is what the cut table the splitter walks holds
--mw-save <file>	the words this run found, for the chunks of a split run to share
-a, --omen-alphabet <n>	OMEN alphabet size, 2 to 255 (default 100)
--omen-alphabet-save <file> / --omen-alphabet-file <file>	the byte frequencies the OMEN alphabet is chosen from
-M, --memory-max <n>	a ceiling on everything that grows with the corpus. Without it, a figure read off free memory decides when the counters spill
-A, --admission-min <n>	a terminal earns a slot only once seen n times, decided while counting, so what is below n never takes memory: a bound on the run rather than on the ruleset, 1 to 15. MANUAL.md section 5.1 sets it against -f
-n, --omen-ngram <n>	OMEN n-gram size, 2 to 8 (default 4)
--omen-level-max <n>	deepest OMEN keyspace level, 1 to 81. Past it the escape generates nothing. Left alone, each ruleset is cut at the level that first reaches 1e15 guesses, which is what decides the memory hashcat holds for it
-J, --df-disable	keep the lines the filters take for hashes, keys and tokens
-T, --threads <n>	threads counting (default: the machine's cores)
-v, --verbose	report what each pass costs
--version	the version, and the ruleset format it writes

What it says when it is done

The last line of a run is what that ruleset will cost the side that reads it:

hpcfg: 12.888.098 lines, 29.204.237 passwords, 5.698.352 terminals, 330.266 OMEN contexts
hpcfg: OMEN cut at level 18, 9.46e+15 guesses, 2.66 GB of tables in hashcat

hashcat holds one u64 per length, per level and per OMEN context, and that is the figure. It is worth reading before choosing a wider --omen-ngram: the same corpus at an n-gram of 5 asks for 17.58 GB. MANUAL.md has the formula and what the defaults were measured against.

Looking at what came out

./hpcfg -i /tmp/ruleset                          # what it holds, and what is wrong with it
./hpcfg -i /tmp/ruleset --ruleset-diff /tmp/other   # how much of the mass the two agree on

For a reader other than hashcat

./hpcfg -t rockyou.txt -g /tmp/ruleset --ruleset-format pcfg-cracker

Training a corpus in pieces

A corpus too large for one run is trained in parts and folded together. The parts have to share the word set and the byte frequencies, or each one chooses its own and the pieces do not add up to what a single run would have given. So the first pass reads the whole corpus for those two files and writes no ruleset - leave -g off:

./hpcfg -t all.txt --mw-save words.txt --omen-alphabet-save bytes.txt

./hpcfg -t part1.txt -g rules.1 -m words.txt --omen-alphabet-file bytes.txt
./hpcfg -t part2.txt -g rules.2 -m words.txt --omen-alphabet-file bytes.txt

./hpcfg --rulesets-merge rules.1 --rulesets-merge rules.2 -g rules.all -t all.txt

--rulesets-merge is repeated once per ruleset, -g names the output, and -t hands the merge a corpus to take the escape's level distribution from, which the rulesets cannot carry between them. --rulesets-merge-weights 1:2 sets the shares by hand instead of taking each ruleset's own size.

That merge reproduces training on the union: the result above is identical to ./hpcfg -t all.txt -g rules.one -m words.txt --omen-alphabet-file bytes.txt in every file but config.ini, which carries a fresh uuid and the names this run was given.

Two things have to hold for that: the counters must not have been bounded by -M or --admission-min, and the escape must divide evenly. MANUAL.md says what each of those costs when it does not hold.

The flag names read as a domain, then the subject, then the qualifier last. -h lists the rest.

tests/hpcfg_test.py holds the checks. It needs Python and nothing else, builds every corpus it uses, and runs the same on Windows:

python tests/hpcfg_test.py                 # 101 tests, five of them skipped
python tests/hpcfg_test.py --list          # what each one checks
python tests/hpcfg_test.py -k merge        # only tests whose name contains this

The five skipped ones ask another implementation to open a ruleset hpcfg wrote, and are the ones worth running before a release, because they are the only ones that check hpcfg against something other than itself. MANUAL.md has the full invocation.

--full builds hpcfg under AddressSanitizer, ThreadSanitizer and UndefinedBehaviorSanitizer and runs the suite against each. It is not in CI: it needs a compiler per sanitizer and takes about ten minutes.

Three tables under src/ are generated from someone else's data - Go's Unicode tables, xkeyboard-config's symbol files, and IANA's list of top level domains - and committed, so building needs neither Go nor Python. How they are regenerated, and what the workflow checks about each, is in MANUAL.md.

State

Trains, counts on every core, and writes a ruleset hashcat reads, OMEN included. Bounded by a memory budget and an admission filter rather than by hope. Rulesets can be folded together afterwards with --rulesets-merge, which is what lets a corpus be trained in pieces. Not yet done: the extended ruleset.

Two departures from the implementations it descends from are deliberate, and they are the same decision twice: a multiword set completed before any password is parsed against it, and a flat table in place of the trie. Both are what make training deterministic - the same corpus trains to the same ruleset, byte for byte - and determinism is what lets --rulesets-merge reproduce a ruleset from its parts.

Under --ruleset-format pcfg-cracker the ruleset is one pcfg_cracker and pcfg-go load and hashcat opens, and under --ruleset-format cpcfg it is one Cpcfg loads in every mode it has - which is checked rather than claimed, and checked again at every tag, because all four of them move.

In detail

MANUAL.md is the rest: what the tokenizer decides and why, what the OMEN defaults were measured against, what bounds the memory on both sides of the handoff, and how the generated tables are kept honest.

Measurements

How hpcfg compares with the implementations it descends from - the numbers, the commits each was taken against, and the defects the comparison found - is in COMPARISONS.md, one section per tag and appended to rather than rewritten.

What changed

CHANGELOG.md is what moved between one tag and the next, and what a caller would notice - including the option names that were removed.