The mibcorpus tool

mibdump compiles the modules it is given. Building a corpus is a different shape of job: it takes a declared set of source namespaces – a vendor’s directory of ASN.1 files, the standard modules pysmi bundles – and produces every artifact the corpus publishes, with the property that two runs over the same inputs produce the same bytes.

mibcorpus is that job, and pysmi.corpus is the library beneath it.

$ mibcorpus --help
Synopsis:
  Build a MIB corpus from a declared set of source namespaces
Documentation:
  https://github.com/pysnmp/pysmi
Usage: mibcorpus [--help]
      [--version]
      [--quiet]
      [--debug=<all|borrower|codegen|compiler|grammar|lexer|parser|reader|searcher|writer>]
      [--manifest=<FILE>]
      [--namespace=<TIER>:<NAME>:<SOURCE>]
      [--output-directory=<DIRECTORY>]
      [--frozen-index=<FILE>]
      [--parse-cache=<DIRECTORY>]
      [--emit=<ARTIFACT>[:<PATH>]]
      [--corpus-version=<VERSION>]
      [--corpus-id=<NAME>]
      [--no-bundled-mibs]
      [--fail-on-errors]

The input set

A corpus is built from source namespaces, declared in a manifest:

{
  "version": 1,
  "namespaces": [
    {"name": "standard", "source": "package:pysmi.mibs.asn1",
     "tier": "standard"},
    {"include": "src/vendor/*", "tier": "vendor"}
  ]
}

Each namespace has a name, a source and a tier. The source is a directory, or package: and a dotted package name for the modules a Python package ships. An include entry stands for every directory matching a glob, expanded in sorted order, so a corpus of three hundred vendor directories does not have to list them by hand and still gets them in one order. Paths are relative to the manifest’s own directory.

The tier is one of standard, draft and vendor, and it is what tells the OID index that a standard module owns an arc a vendor module also defines. It is declared rather than inferred from where a file sits, because a tier is a statement about a source, and the source is what the manifest names.

A namespace may also declare "publish": false. It is then a resolution source: it supplies what the published modules import and contributes nothing to the output. Nothing of its is staged, compiled or indexed, and it is stubbed in every output format so that it cannot arrive as a dependency either. On the command line that is --resolve-namespace rather than --namespace.

The order matters. Two namespaces can hold a module of the same name and exactly one copy of it can be published; the newest MODULE-IDENTITY revision wins and source order breaks the tie, which is the rule compile() documents and applies.

A corpus of a few modules

"publish": false narrows a build by namespace. --publish-only narrows it by module, and everything outside the selection becomes a resolution source in exactly the same sense: read, available to resolve an import, and carried nowhere.

mibcorpus --manifest=corpus.json --output-directory=preview \
  --emit=site --emit=report \
  --publish-only=CISCO-IPMCAST-MIB \
  --publish-only=TEL2N-MIB

This exists for a question a namespace cannot express: what do the modules this change touched look like? Neither obvious answer works. A build over the changed files alone fails on every import they make – a vendor module that imports SNMPv2-TC needs SNMPv2-TC to be there. A build over the whole corpus answers in five minutes and buries the two pages that were asked about in seven thousand that were not. So the input set stays whole and the output set narrows.

What makes the result worth looking at is that it is the same corpus:

module built narrowed  ==  module built whole

byte for byte, in the ASN.1 and in the JSON, because it is the same compile over the same resolved sources. This is the property "publish": false rests on as well, and tests/test_corpus_select.py holds the narrowed build to it.

--publish-only-from=FILE reads the same names from a file, one per line, blanks and # comments ignored, for a caller that computes the set rather than typing it.

Two refusals, both because the quiet alternative is worse:

• A selected module no publishing namespace holds is an error naming every one that missed. A caller that named three modules wants three, and a preview that silently publishes two has answered a question nobody asked. A namespace holds what its files declare, not what they are named, so a file too damaged to lex a module header out of holds no module and is reported here rather than as an empty page.

• An empty selection is an error rather than a whole corpus. Naming no selection at all is a distribution build; computing one and computing it empty is a request for a corpus of nothing, and the two must not collapse into each other – a pull request that changed no MIB would otherwise publish the whole corpus.

report.json records what was asked for under selected:

{
  "selected": {
    "requested": ["CISCO-IPMCAST-MIB", "TEL2N-MIB"],
    "published": ["TEL2N-MIB"],
    "failed": ["CISCO-IPMCAST-MIB"]
  }
}

which is what tells a check that a module was asked for and did not compile, as against never having been asked for. --fail-on-errors turns that into a non-zero exit.

What the corpus is

A manifest also says which artifacts the corpus carries and what must be true of the result. Without those two keys the definition of a distribution is split between a JSON file pysmi reads and a build script pysmi has never seen, and a second publisher reproducing the corpus starts by copying the build script – which means copying decisions that were never theirs.

{
  "version": 1,
  "emit": ["asn1", "index-v2", "report"],
  "expect": {
    "modules": {"min": 8000},
    "failures": {"max": 12},
    "namespaces-present": ["iana", "ietf"]
  },
  "namespaces": [
    {"include": "src/vendor/*", "tier": "vendor"}
  ]
}

emit takes the same artifact names --emit does, a path of its own included – "json:build/jsondoc". --emit on the command line overrides it, as a flag overrides a file. With it declared, mibcorpus --manifest=M --output-directory=D is the whole build, and the reason a corpus carries what it carries sits next to the set itself.

expect is checked after the build, against the same numbers report.json records:

Expectation	What it bounds
modules	How many modules the corpus publishes, counted once however many namespaces hold one. Independent of the emit set.
failures	How many of those failed to compile, counted once however many output formats they failed in.
namespaces-present	Namespaces that must be in the build and published.

modules and failures take min, max or both. An expectation missed prints the field, the bound and the actual value, and exits 65. An expectation the build does not report – a misspelling among them – is refused when the manifest is read, since one silently ignored leaves the publisher believing a check is being made that never is.

What it produces

Artifact	What it is
asn1/	One file per module name, flat, named for the module. Staged from what the compile resolved, so the ASN.1 beside a compiled module is the text it was compiled from. Its naming is a contract – see How the ASN.1 tree is named.
notexts/	pysnmp modules without DESCRIPTION and the other texts.
texts/	pysnmp modules with them, in their original layout.
json/	jsondoc documents. Without DESCRIPTION and the other texts unless the build asks – see Prose in the JSON.
index-v2.csv	The ranked OID index: for every OID, the one module that owns it.
index.csv	The legacy OID index, which replays --frozen-index so that consumers keying on the module an OID resolves to keep the answers they already have.
standard.txt	The modules from every standard namespace, less the RFC* and SNMPv2* prefixes the published file has never carried.
closure.json	Per module, the files a consumer needs in order to load it, and anything it needs that the corpus does not hold. See The files a module needs.
entity.json	Which enterprise arcs the corpus registers under, who the registry says holds each, and who to report a problem to. Asked for by name, and wants --oid-registry. See Who registered an arc.
arcs.json	What every arc the corpus reaches is called, and which authority says so. Asked for by name. See What an arc is called.
report.json	What the build did: the failure inventory, the modules more than one namespace holds, the node counts, which JSON implementation wrote the artifacts, and how long each phase took.

The two databases and the site are not in that layout and have to be asked for by name:

Artifact	What it is
core.db	The corpus database: every node the corpus defines, keyed for lookup by OID and by name, and ordered for GETNEXT. See Corpus database schema. Building it costs a pass nothing else needs, which is why it is opt-in rather than part of the default layout.
search.db	The same schema at a seventeenth the size: everything but node and the type table it references, for a consumer that needs to know which module and never what the object is. 15 MB against core.db’s 256 over the published corpus, which is what makes it something a site can publish. See Corpus database schema.
site/	The corpus as pages: one per module, per registrant and per node of the registration tree. Asked for by name, since a corpus published as files rather than as a site has no use for it. See The browsable site.

--emit narrows this to the artifacts named, so a build can ask for just the index or just the JSON.

Both databases and the two indexes are projections of the jsondoc tree, which is a dependency of pysmi’s own rather than of the caller’s: a build asking for one of them without asking for json gets a tree staged in a temporary directory and removed when the build ends, raise or return. The corpus carries what was named and nothing else.

The site and the entity index read a tree with the texts in it. A module page carries each definition’s DESCRIPTION, and a registrant is named from the module’s own ORGANIZATION and CONTACT-INFO before the registry is consulted; JsonCodeGen gates all three behind one switch, so a lean tree holds none of them. A build rendering either gets a texts-carrying tree staged the same way – even where it named a lean one, since the lean tree cannot answer. That costs a render pass rather than a parse, since the parse is shared, and the tree the build publishes is untouched: a publisher asking for a lean json still gets a lean json.

Over pysnmp/mibs the prose is 90.2 MB of the module pages, 85% of definitions carrying one. Emit json-texts to keep the tree that produced it.

mibcorpus --manifest=corpus.json --output-directory=output \
    --emit=asn1 --emit=index-v2 --emit=core-db

Emit json to keep the tree, and give it a path of its own to say where it goes.

Every artifact but report.json is byte-reproducible. The report is the build’s log and records elapsed time.

One build, more than one tree

A distribution is not always one tree. The same corpus goes out as the data a runtime fetches and as the pages a reader browses, and those may belong on different hosts with different limits. Building it twice parses every module twice to reach the same answer.

A manifest declaring publications instead of emit says a build writes several trees:

{
  "version": 1,
  "namespaces": [{"include": "src/*", "tier": "vendor"}],
  "publications": [
    {"name": "data",  "emit": ["asn1", "json", "index-v2"], "output": "data"},
    {"name": "pages", "emit": ["site", "search-db"],        "output": "pages"}
  ]
}

Each publication names what it carries and where it goes, under --output-directory. The corpus is parsed once for all of them: the parse cache is shared across the plan, and parsing is about three quarters of a pass.

emit and publications are alternatives, not layers. A manifest declaring both is refused, because the artifacts would belong to no tree in particular and a reader could not tell which by looking. A publication naming no output writes into the build directory itself, which is the ordinary single-tree build said the long way.

--emit on the command line collapses the plan to the one tree it names, as a flag overrides a file everywhere else here.

Every publication is summarized and held to the manifest’s expect: a build that wrote two trees and checked one has not checked the build.

Keeping the parse trees

--parse-cache=<DIRECTORY> keeps parse trees between runs, so a rebuild of a mostly unchanged corpus reparses only what changed. The key is the module’s own text together with the identity of the parser, so an unedited module keeps its entry and an edited one cannot match a stale tree. Without it the trees are held in memory and discarded with the process, which is right for a one-off build and wasteful for a nightly.

It stores pickles, and reading one reconstructs arbitrary Python objects. Name a directory the build owns, never one written by anything you would not run, and never one shared across trust domains. A damaged, truncated or unreadable entry is a miss rather than an error: the cache is an optimisation and never a source of truth.

How the JSON is written

json/ and the JSON index are written compactly – one line per document, no space after a separator, UTF-8 written as itself rather than escaped. They are generated trees that nobody reads by eye, and indenting them costs about 30% of the tree on disk: roughly 50 MB over a corpus the size of the one pysnmp/mibs publishes. Gzipped the difference is about 8%, because gzip already eats indentation, so this is a saving on a checkout, an image and a Pages site rather than on bandwidth.

report.json is not written that way and stays indented. It is the build’s log, read by a person looking at a failed build.

A faster JSON implementation is used when one is installed:

pip install 'pysnmp-pysmi[fast]'

That pulls in orjson; msgspec is taken instead if a caller already has it. Either is worth about 4% of a build, because SMI parsing dominates everything else, which is why it is an extra rather than a dependency – 4% does not buy a wheel with a compiled extension in it for everyone.

What is installed does not change a published byte. The standard library is the reference, the faster encoders are used only where they agree with it, and tests/test_jsonio.py asserts that agreement over every document the bundled corpus produces rather than trusting that it holds. A build with the extra and one without are the same corpus. report.json records which implementation ran, as json, so two builds being compared can each say what they resolved.

The files a module needs

core.db’s import table holds the direct edges – this module imports that symbol from that module. The question consumers actually arrive with is the closure: which files do I need in order to load this module? closure.json answers it per module:

{
  "meta": {"schema": 1, "modules": 5347, "incomplete": 0},
  "closure": {
    "IF-MIB": {
      "files": ["IANAifType-MIB", "IF-MIB", "SNMPv2-CONF",
                "SNMPv2-MIB", "SNMPv2-SMI", "SNMPv2-TC"],
      "missing": []
    }
  }
}

(Re-indented to be read; the file is written on one line.)

The build resolves every one of those edges in order to compile, so having each consumer re-walk a table to recover a fact the compiler established is the pattern this exists to avoid. Three arrive with the question: a page answering “the files you need”, anyone packaging a subset – an air-gapped install, an image carrying only what one product needs – and pysnmp, deciding what to preload.

Two things the artifact settles rather than leaving to the caller:

• A module is in its own closure. IF-MIB needs six files and one of them is IF-MIB. That makes the list directly usable as a file list, which is what most callers want.

• A dependency the corpus does not hold is recorded, not dropped. It goes in missing rather than quietly out of files, so a caller can tell “this module needs nothing else” from “something it needs is not here”. A module is listed however deep it was reached from: a hole anywhere below a module is a hole for that module, which still cannot be loaded. The build report counts them, as closure.incomplete.

The names are module names, which are the names of the files in asn1/ – see How the ASN.1 tree is named – so a caller holding a closure holds the file list and can build the URLs itself.

Prose in the JSON

A published jsondoc carries names, OIDs, syntax, access and status, and no prose. IF-MIB.json has no description on ifOperStatus, though the module’s text describes all seven of its enumerated states. Measured over a 300-file sample, DESCRIPTION, REFERENCE and CONTACT-INFO are 38% of the text of a MIB – a large part of the module the JSON rendering omits, and a consumer that wants it has to fetch and parse the ASN.1, which means a second SMI parser for prose the compiler already read.

json-texts is a jsondoc tree with the texts in it:

mibcorpus --manifest=corpus.json --output-directory=output --emit=json-texts

That writes json/ – one tree, carrying the prose, at one compile pass.

It is a destination of its own rather than a flag on json, so a build may also have both:

mibcorpus --manifest=corpus.json --output-directory=output \
    --emit=json --emit=json-texts:build/full

which publishes the lean tree and keeps a complete one for something that needs the prose – a site generator rendering descriptions into its pages, say – without the published artifact growing. That is two passes, because it is two trees. Naming both at one path is refused: the second pass would overwrite the first, and which of them survived would depend on the order the emit list was read in.

The texts cost roughly 70% more on disk, which over a corpus is a tree half again as large. That is a decision for the build, which is the argument for asking rather than assuming, and json-texts is not in the default layout.

What genTexts gates is more than descriptions: JsonCodeGen.gen_module_identity puts organization and contactinfo behind the same switch. CISCO-ENTITY-ALARM-MIB carries a full CONTACT-INFO block in its ASN.1 – Cisco Systems, Customer Service, a postal address, a phone number and cs-snmp@cisco.com – and none of it reaches the published JSON. Over a 500-module sample, 90% of modules carry ORGANIZATION and CONTACT-INFO, and 66% of those carry an email. That is the publisher’s own statement of where to report a problem, and it is what The mibcorpus tool cannot show a reader until this is turned on.

keepTextsLayout is not turned on with it. The two are separate for the pysnmp destinations and stay separate here: a JSON consumer generally wants the text normalised rather than the publisher’s line breaks preserved.

Who registered an arc

A corpus knows that CISCO-ENTITY-ALARM-MIB registers under 1.3.6.1.4.1.9. It does not know that 1.3.6.1.4.1.9 is Cisco: nothing in the MIB text says so in a form anything can rely on, and the directory a file sits in is a filing convention rather than a registration. Over pysnmp/mibs the two disagree in practice – the aironet directory holds Cisco modules, and src/vendor/cisco/ALTIGA-* registers under Altiga’s arc.

The registration is a published fact. --oid-registry supplies it and --emit=entity writes the projection:

mibcorpus --manifest=corpus.json --output-directory=output \
    --emit=entity --oid-registry=pen-snapshot.csv

{
  "meta": {"schema": 1, "arcs": 351, "named": 350, "unregistered": 1},
  "entity": {
    "1.3.6.1.4.1.9": {
      "number": 9,
      "organization": "Cisco Systems, Inc.",
      "modules": ["CISCO-AAA-CLIENT-MIB", "..."],
      "contacts": [
        {"source": "module", "organization": "Cisco Systems, Inc.",
         "contact": "Cisco Systems\n Customer Service\n ...",
         "email": "", "module": "CISCO-ENTITY-ALARM-MIB", "authority": ""},
        {"source": "registry", "organization": "Cisco Systems, Inc.",
         "contact": "Dave J", "email": "davej&cisco.com", "module": "",
         "authority": "https://www.iana.org/assignments/enterprise-numbers#9"}
      ]
    }
  }
}

Over pysnmp/mibs the distinct enterprise arcs outnumber the vendor directories by a fifth, and the registry names all but a handful of them. That gap is the argument for driving navigation from the registry rather than from the tree; report.json carries the run’s own counts under entity.

An arc the registry does not name is reported as unregistered, never guessed at. pysnmp/mibs has exactly one, 1.3.6.1.4.1.1004849, above anything IANA has allocated. The build report counts them, as entity.unregistered.

The index groups by arc, not by company. The registry maps arcs to registrants and a company can hold several: Cisco modules sit under 1.3.6.1.4.1.9 and, from the Altiga acquisition, under 1.3.6.1.4.1.3076, which IANA still lists as “Altiga Networks, Inc.”. Nothing in the registry models an acquisition and PySMI does not infer one.

entity.json is not in the default layout: it is only worth having with a registry to name its arcs, and that is an input the caller supplies.

Owner contact

Each arc carries who to report a problem to, best source first, each naming its source so a reader can weigh a vendor’s current support address against an undated registration:

1. The module’s own ``CONTACT-INFO`` – a corporate block with a role mailbox, and the publisher’s current statement of where to report a problem. It reaches the corpus only when the build carries texts, since JsonCodeGen gates organization and contactinfo behind the same switch as description – see pysnmp/pysmi#277. Where an arc has many modules, the one with the newest LAST-UPDATED speaks for it, with the module name breaking a tie so that two builds agree.

2. The IANA registration – registrant, contact name, contact email. Second because it is undated and often stale, and carrying the authority link, because a correction to a registration belongs at IANA.

Each is rendered as its source publishes it. An email from the registry is davej&cisco.com because that is what the registry says; a module’s block is reproduced whole rather than picked apart for an address, since a block holds a company, a postal address, a phone number and a mailbox and choosing between them is not the build’s decision.

Remediation is precedence rather than a suppression list. A registrant who does not want an undated personal registration standing as the contact for their arc publishes a module carrying current ORGANIZATION and CONTACT-INFO, and source 1 displaces source 2 on the next build.

The registry file

Taken as an input. Never bundled, never fetched. A corpus build resolves nothing over the network – a build with the network unplugged produces the same corpus as one without – and a registry that changes daily, fetched at build time, would end that. 5.1 MB that changes daily is also not a thing to vendor into a compiler.

--oid-registry is repeatable and reads two registries: the Private Enterprise Numbers registry, in either the published four-line-record format or a reduced CSV, and IANA’s smi-numbers XML (see What an arc is called). Which one a file is comes from its content rather than from its name, since a snapshot a repository commits is called whatever that repository calls it. A repository that keeps a snapshot reduces a download once:

python -m pysmi.registry enterprise-numbers.txt > pen-snapshot.csv

The reduced form keeps the whole registry, whole records, by default. --fields narrows the record and --only/--only-from narrow the rows. What leaves IANA’s copy is the committing repository’s decision rather than PySMI’s, so both are arguments rather than a hard-coded projection. Nothing is normalised: the output is a rendering of IANA’s record rather than a corrected version of it.

The registry runs to tens of thousands of registrations and IANA revises it daily, so a repository committing all of it commits a large file and re-diffs the whole of it every month. The other choice is a snapshot of the registrants this corpus’s own arcs use, which is a few hundred rows rather than the whole registry:

python -m pysmi.registry --only-from=arcs.txt enterprise-numbers.txt \
    > pen-snapshot.csv

--only-from reads enterprise numbers one per line, blanks and # comments ignored, which is what a corpus can write out of its own index. The cost is staleness – a module arriving later under an arc the snapshot predates goes nameless – and the build says when that has happened rather than leaving it to be found on the site.

What an arc is called

The OID index ranks modules to decide which one owns an arc. The rule is total and works for arcs a module actually registers; for the arcs above those it has nothing good to choose from, so it picks whichever module happened to mention the arc on its way somewhere else. Measured against pysnmp/mibs:

arc	is	attributed to
1.3	identified-organization	OCCAM-ETHERLIKE-MIB
1.3.6	dod	OCCAM-ETHERLIKE-MIB
1.3.6.1.6	snmpv2	RAPID-CITY
1.3.6.1.6.3	snmpModules	RAPID-CITY
1.2	ISO member-body	IEEE802dot11-MIB
0.0	ITU-T recommendation	DLSW-MIB

A tree that says snmpModules belongs to a Nortel enterprise MIB is wrong in a way that matters, and no ranking over MIB text can fix it, because the fact is not in the MIB text. Twenty-five arcs are claimed by nothing at all, and a tree still has to render a path through them – 1.3.6.1.4.1.9 among them, since no module registers Cisco’s bare arc, only what hangs beneath it.

--emit=arcs writes the answer, from the registries the build was given:

mibcorpus --manifest=corpus.json --output-directory=output --emit=arcs \
    --oid-registry=smi-numbers.xml --oid-registry=pen-snapshot.csv

{
  "meta": {"schema": 1, "arcs": 6694,
           "by-source": {"module": 5900, "registry": 700,
                         "standard": 15, "unnamed": 79}},
  "arc": {
    "1.3.6.1.6.3": {"name": "snmpModules", "source": "registry",
                    "reference": "https://www.iana.org/assignments/smi-numbers"},
    "1.0.8802": {"name": "iso8802", "source": "standard",
                 "reference": "ISO-IEC 8802"}
  }
}

Every name says which kind of fact it is

source is rendered, because an IANA registration, a cited standard and a name read off whichever MIB mentioned an arc are three different things and a reader has to be able to tell them apart. Strongest first:

registry

IANA. smi-numbers names the 1.3.6.1 subtree – and, through the dotted names in its own registry descriptions, 1 iso down to 1.3.6.1 internet, which nothing else in the file names. The Private Enterprise Numbers registry names a bare enterprise arc.

standard

A standard, cited. ITU-T’s OID registry does not answer, and IEEE publishes landing pages and PDFs, so there is no feed for 0, 1.0, 1.2, 1.3.111 or 1.0.8802 – the chains IEEE registers its 802 MIBs under, and LLDP-MIB sits at 1.0.8802.1.1.2. Those are defined in ITU-T X.660 and ISO/IEC 9834-1 and have not changed in decades, so PySMI carries a small table with a reference per entry. A cited name is not a registration and is labelled as what it is.

module

A module’s own descriptor, which is what the index has always used. Still here, still useful, and now labelled rather than presented as though it were an authority’s answer.

An arc nothing names says so, with an empty name rather than a borrowed one. The build report counts them.

When a committed PEN snapshot has gone stale

A repository that commits the whole Private Enterprise Numbers registry commits tens of thousands of registrants and re-diffs all of them every time IANA moves, which is daily. Committing only the registrants its own arcs use is a far smaller file and a far smaller monthly diff, at the cost of going stale: the first module the corpus gains under a newer enterprise arc has no registrant in the snapshot, and that registrant’s page renders nameless.

Nothing about the build would otherwise say so – the arc is still in the index, still reachable, still rendered, just blank. So the build says it:

WARNING  3 enterprise arc(s) no PEN registrant names: 62373, 99999, 100001

and report.json carries the count as arcs.unregistered-enterprises, so a build nobody watched can still be asked. arcs.json carries every one of them, since a log line is not the artifact.

It is a warning and never a failure. An arc can be registered to nobody, and IANA’s registry has gaps of its own. An arc a module names is still counted: the module’s own descriptor says what the vendor calls its subtree, not who registered it, and it is the registrant a refresh would supply.

pysmi.corpus.arcs.unregistered() is the same answer as data, for a build that wants to act on it.

What is in the inventory

The arc set comes from the OID index and every prefix of it – the registration tree – rather than from every OID a module defines, which is tens of thousands of arcs against one per defined OID. The difference is objects, and an object’s arc is a thing inside a module rather than a node anybody navigates to.

It is not filtered to one subtree. Nearly every index row of a vendor corpus sits under 1.3.6.1.4.1 and most of the rest under 1.3.6.1.2.1, but IEEE publishes its 802.1 MIBs under 1.3.111.2.802.1 and LLDP-MIB registers under 1.0.8802.1.1.2. A 1.3.6.1 filter drops both, and LLDP is among the most widely polled MIBs there is. Arc depth runs from 2 to 20.

That inventory counts an arc whether or not anything would render a page for it. Which of them a site gives a page to is a narrower question, and pysmi.corpus.pages answers it: the arcs above the modules, which is a small fraction of the arc index and two orders of magnitude below a page per defined OID. An arc at a module’s anchor is the module, and an arc below one is an object the module page already renders.

The browsable site

--emit=site writes the corpus as pages, from the same jsondoc tree the indexes and the database are projections of. Three trees and an entry point, named so that nothing collides with a corpus path:

mib/<MODULE>/	one module: identity, provenance, the repair and its reason, the load order, imports, what imports it, the arcs below it, and every object, notification, textual convention and conformance statement with its OID, syntax, access, status and description
entity/<PEN>/	one registrant: who the registry says holds the arc, who to report a problem to, and every module the corpus holds under it
oid/<arc>/	one node of the registration tree, and its children
browse/	the entry point: what the corpus holds in figures, the most recently revised modules of each tier, and the module list

mibcorpus --manifest=corpus.json --output-directory=output \
    --emit=site --emit=json \
    --oid-registry=smi-numbers.xml --oid-registry=pen-snapshot.csv \
    --site-name="pysnmp/mibs"

Over a corpus of thousands of modules that is thousands of pages and a few hundred megabytes, in seconds rather than minutes on top of the build that produced the corpus.

What the entry point states

browse/ leads with the module count, then a tile per figure the build already computed: modules per tier, enterprise arcs registered under, nodes in the arc index, objects, notifications and textual conventions defined, modules served with a patch applied to the publisher’s text, and modules whose import closure names something the corpus does not hold. A figure of zero gets no tile, so a build given no arc registry states nothing about arcs rather than stating none.

Under the figures, one table per tier – standard, Internet-Draft, vendor – gives the most recently revised modules of that tier, by the LAST-UPDATED or REVISION date the module itself carries. Split by tier because a corpus of thousands of vendor modules and a few hundred standard ones has no recent standard revision in a single list, and the standard modules are what a reader resolving an OID is looking at.

What the distribution changed, beside what the publishers did

Every date in a MIB is its publisher’s, so those tables answer what has the industry published lately and cannot answer what has this distribution done lately. A module taken today may carry a revision from decades back, and a module corrected today carries whatever date its publisher last set.

That second date is a fact about a distribution rather than about a MIB, so it arrives from one. site.changed names a JSON file of module to YYYY-MM-DD and what to call it:

{
  "site": {
    "changed": {
      "label": "Added or updated here",
      "dates": "output/module-dates.json"
    }
  }
}

{"IF-MIB": "2026-09-17", "CISCO-IPMCAST-MIB": "2026-04-02"}

Given one, the entry point offers both orderings as tabs – the publishers’ revision dates and the distribution’s own – and a reader switches between them without a fetch and without a script: each tab is a label pointing at a radio, and the stylesheet shows the panel whose radio is checked. Both panels are in the markup either way, so a crawler reads all of it and a reader with no CSS gets the two lists one after the other under their own headings.

How a distribution knows the date is its own business. pysnmp/mibs reads the commit that last touched each module’s file, which is close enough to “when we published it” for a repository that commits what it publishes.

A module the file does not name has no such date and is listed under none, which is what lets a distribution date the part of the corpus it tracks and stay quiet about the rest – pysnmp/mibs tracks the modules in its own tree and takes the standard ones from pysmi, so the standard tier has no such date and gets no such table. The note under each table counts the pool it came from for that reason – “of the vendor modules this distribution tracks” rather than “of the ones this corpus holds”, which over a corpus whose standard tier comes from elsewhere are different numbers and different claims.

A date the lists cannot sort by – anything but YYYY-MM-DD – fails the build rather than reaching a page, and so does a named file that will not read. A site that quietly published an empty list here would state that the distribution has changed nothing, which is a different claim from “this build was not told”.

Every figure is a sum over the module pages the build renders, taken as it renders them, so the entry point costs no second pass over the corpus. The figures are in the bytes like everything else here: the front page is the first thing a crawler reads.

Where the repairs come from

A module page says whether the text served here differs from the publisher’s, and why. That panel is filled by --patch-directory:

mibcorpus --manifest=corpus.json --output-directory=output \
    --emit=site --patch-directory=scripts/mib-patches

The directory is read recursively and the file name before .patch is the module, wherever in the tree it sits – a corpus large enough to file its repairs per vendor keeps scripts/mib-patches/cisco/CISCO-IPMCAST-MIB.patch and this reads it. Two files naming one module is an error: two repairs for one module are two different opinions, and applying whichever the filesystem offered second is how a corpus comes to publish text nobody chose.

Nothing is applied. The sources are read exactly as they stand; this is a description of a repair already in them, which is the arrangement The mibpatch tool describes – the checked-in text is the publisher’s text plus the patch, so the patch is the record of the difference rather than a step in the build.

Both halves of the header reach the page. The Defect: lines become links into whichever catalogue the patch cites – Defects a patch repairs for pysmi’s own identifiers, a downstream catalogue for a downstream corpus’s – and the note under them stays with the diff, because the catalogue says what the defect is and only the note says why this module has it.

Why the generator is here

The site is a function of a corpus, so an enterprise pointing mibcorpus at its own manifest – a private collection, a downstream project vendoring additions – gets the same site over its own modules. The alternative is a second implementation in whichever repository publishes the corpus, which is the shape pysnmp/mibs#365 removed: that repository’s compiler, dependency resolver and OID indexer were each a second implementation of PySMI’s, and each had drifted.

The module set is the corpus

Pages are written for what the manifest declares and nothing else. A directory sitting beside the sources is not a namespace, so it is not in the corpus and gets no pages; and a namespace declared for resolution only – publish false, which is how a compact corpus resolves against the standard modules without carrying them – contributes none either.

Everything is in the bytes

No page fetches anything: not the object table, not the description, not the navigation. #284 measured the alternative and it costs the same on disk, while being unreadable to the AI crawlers that do not run JavaScript – to those, a client-rendered object table is a page about a MIB module that does not say what the module defines.

Links are relative to the site root, so a site published under a path on a project host works without being told where it lives, and so does one opened from a local directory.

The page count is bounded

Three rules keep it that way, and each is its own module:

• the OID tree stops at the modules, which pysmi.corpus.pages decides. An arc at a module’s anchor is that module and resolves to its page; an arc below one is an object the module page already renders in context.

• long lists split by range key rather than page number (Splitting a long list page), so adding one module does not renumber every page after it. page-size sets how long a list gets first, as one number for every list or as an object naming them – {"browse": 500, "entity": 200}. The module list is the site’s front door and wants few pages; a registrant’s list is reached by somebody already narrowed to one vendor, and the largest vendors hold enough modules that the same size gives them several pages of 500 links each.

• the list of modules importing a given one is capped. It is the one list on a module page with no natural bound: a large fraction of a corpus imports IF-MIB, and nearly all of it imports SNMPv2-SMI. The count is the fact worth stating.

No page is an orphan. The case that breaks this is a structural arc whose parent is a module anchor: the parent has no arc page, so only the module page can link down to it, and it does – see Under this module’s arcs.

The crawl surface

Declaring a base-url says this build is producing a distribution site rather than a subtree somebody else will assemble, and the whole crawl surface is written with it: a canonical link and a meta description per page, JSON-LD, sitemap.xml as an index, robots.txt and llms.txt. Without it there is nothing to put in a canonical link, and guessing an origin would publish a site claiming to live somewhere it does not.

data-url is the second origin, for a distribution that serves its pages and its downloads from two hosts. This one does: a page URL is a directory that a host resolves to index.html, and object storage serves the key it is given, so the pages and the files cannot share an origin without something on the serving path rewriting every request. Only the llms.txt bulk links read it. A canonical link and a sitemap entry describe a page, and a page is always on base-url. Omitted, one origin serves both.

{
  "site": {
    "base-url": "https://mibsdepot.com",
    "data-url": "https://data.mibsdepot.com",
    "name": "MIBs Depot",
    "description": "SNMP MIB modules from their publishers.",
    "crawl": {
      "*": {"disallow": ["asn1", "json", "index-v2", "core-db"]},
      "ClaudeBot": {"allow": ["asn1", "json", "index-v2"]}
    }
  }
}

A crawl policy names artifacts, not paths. A distribution saying “keep search engines out of the raw JSON” should not have to know whether that tree is called json or something this release renamed. An artifact the policy names that the build did not produce is skipped rather than written: a rule about a tree that is not there would read as though the tree existed.

The policy is per-agent because a search engine and an agent want opposite things from the same tree. To the first, the hundreds of megabytes of asn1/ and json/ are crawl budget spent on files with no indexing value; to the second they are the point, and llms.txt sends it there deliberately. The agent list is configuration because crawler names change faster than releases.

llms.txt is a convention rather than a mechanism – no major AI crawler is documented as consuming it. It is emitted and nothing depends on it. What reaches an AI crawler is server-rendered HTML and a robots.txt that admits it.

PySMI cannot decide where robots.txt lands. On GitHub Pages a project site cannot serve its own: crawlers read it only from the host root, which belongs to a different repository. So the file is produced here and the deployment places it.

lastmod is a date the content states

A corpus rebuilt on every push that stamps every page with today’s date teaches a crawler the field is noise. A module page’s lastmod is the module’s own newest REVISION or LAST-UPDATED, which is the date its content last changed; a page listing modules takes the newest among them.

So no build clock reaches a sitemap, and two builds of one corpus produce the same one. Over thousands of URLs the dates are the publishers’ own, and today’s is not among them. A page describing something with no readable date is listed without a lastmod at all – which says “I do not know”, where the build date would say something false.

A stamp shaped like a date and not one is dropped rather than ranked, for the reason the index ranker gives: HPR-MIB carries 970514000000Z, month 14, which compares above every real date there will ever be.

What the JSON-LD says, and what it costs

A MIB module is a published artifact with an identifier, a publisher and a revision date, which defines a vocabulary of named terms – DefinedTermSet crossed with Dataset. The identifiers are the OIDs, which is the point: an OID is a globally unique identifier that already exists, and a consumer holding one should be able to match it without reading prose.

It is not free. Enumerating every definition a corpus holds adds about 40% to the site. A first cut also carried each term’s description and an inDefinedTermSet back-reference, which took it half again beyond that; both are gone, the descriptions because they are already in the HTML the JSON-LD sits in, and the back-reference because nesting under hasDefinedTerm already says a term belongs to the set.

Theming

--site-template and --site-stylesheet replace the page frame and the stylesheet, so a distribution publishing this beside its own documentation makes it look like the rest of that documentation without forking the generator. $name substitution, and an unknown placeholder renders as itself rather than raising partway through a build of thousands of pages – a page with $oops on it is something somebody sees.

A named file that cannot be read is refused rather than ignored: falling back would publish a whole site in the wrong skin and say nothing.

What a replacement stylesheet reaches

The definition tables carry no element inside a cell. A corpus holds hundreds of thousands of definitions, and a wrapper per cell was 61 bytes a row over them – a quarter of the module pages – so the cells hold their text and the stylesheet reaches them by position:

selector	the column
.defs	a definition table.
.defs.typed	the five-column shape: Name, OID, Syntax, Access, Status. Without typed it is the three-column one: Name, OID, Status.
td:first-child	the descriptor.
td:nth-child(2)	the OID.
td:last-child	STATUS, in either shape.
.desc	the macro a definition came from, and its DESCRIPTION where the build carries texts. The one element still written inside a cell, because it is a block under the name rather than a column.

Each row carries the descriptor as its id, so mib/IF-MIB/#ifTable is a link to one definition. Style :target to show which one was asked for.

Splitting a long list page

A site rendering a corpus has lists too long for one page: every module it holds, every module one registrant published, the children of a wide OID node. pysmi.corpus.buckets decides what the pieces are called.

Not first letters. A to Z is the obvious index and it fails on MIB names, which are dominated by vendor prefixes rather than spread across the alphabet. Over a vendor corpus the letter C takes a third of the modules by itself, on the strength of CISCO- alone, and most letters take almost none. Lengthening the prefix does not rescue it: inside a registrant essentially every name shares one, so no fixed length gives even buckets, and the length that would work differs between the global list and one registrant. --emit=report states what a given corpus splits into.

Not page numbers. page/4/ is stable only while the list is. Adding one module shifts the contents of every later page, so every already-crawled URL past the insertion point serves different content and reports a fresh lastmod. A range key disturbs only the bucket the new entry lands in.

A range key, derived. The sorted list is partitioned into equal-count runs of the page size, and each run is labelled with the shortest prefix distinguishing its ends from its neighbours’ – so the label length falls out of the local density:

AT..CISCO-DIAMETER-SG-C            200
CISCO-DIAMETER-SG-M..CISCO-HC      200
CISCO-HE..CISCO-LICENSE-MG         200
CISCO-LICENSE-MI..CISCO-PRI        200
CISCO-PRO..CISCO-TM                200
CISCO-TN..CISCO-WDS-IDS-C          200
CISCO-WDS-IDS-M..RP                153

The separator is .. rather than an en dash: an en dash percent-encodes to %E2%80%93, which works and is a needless hazard in a string that crawlers, server logs, shell history, spreadsheets and copy-paste all handle. A plain - is ambiguous in names full of hyphens. .. occurs in neither a module name nor an arc number.

The sort is case-sensitive byte order, which is what the labels imply; a display sort that differed would put entries in buckets whose range excludes them. A wide OID node’s children sort numerically instead, and take their numbers whole as labels – the shortest distinguishing prefix of a decimal number is not a number.

Four things the generator needs beyond the split:

locate

Which bucket a name belongs to, including a name the list did not have when the buckets were built. This is what the browser-side navigation resolves against.

successors

A bucket that outgrows the page size splits, which renames a key – the one case where a range URL moves. Static hosting serves no redirects, so the retired key is emitted as a small page carrying a canonical link and a meta refresh to whatever now covers it. retired names those keys.

abbreviate

Every bucket is listed on every bucket page, so any bucket is one hop from any other and crawl depth does not grow with the corpus. That works only if the strip fits, and a label runs to 27 characters. The URL keeps the whole label; the rendering is clipped.

A single bucket

A list no longer than the page size comes back as one bucket with no key: it keeps its unbucketed URL and renders no key strip.

How the ASN.1 tree is named

The names in asn1/ are a contract rather than an implementation detail. pysnmp resolves a dependency by substituting a module name into a source template, and splunk-connect-for-snmp runs that path in production: addMibCompiler() against https://.../asn1/@mib@, compiling ASN.1 per MIB on demand. There is no directory listing and no second guess – the name is the whole of the request.

So:

• A file is named for the module it defines, taken from the DEFINITIONS ::= BEGIN header, not from the file it was read from. A vendor shipping TEST-TC-MIB in tc.mib publishes it as asn1/TEST-TC-MIB.

• The name is the module’s own spelling, not a case variant of it or of the source file’s.

• There is no extension. @mib@ is replaced with a bare module name, so anything appended makes the file unreachable.

• One name per module. A source file defining two modules is published under both names, since either one may be what a consumer asks for.

• A module the corpus only resolves against is not there at all. A namespace declared "publish": false reaches no output tree.

• Only what compiled is there. A module the build could not compile is left out, and so is anything that imports it. Serving its ASN.1 buys a consumer nothing – it fetches the text, compiles it with the same pysmi and fails where this build failed, one round trip later. asn1/, standard.txt, both indexes and core.db therefore carry one module set rather than several. What was dropped is named in report.json, with the error that dropped it.

tests/test_corpus_asn1_contract.py pins this, and the consumer layer compiles a module out of an emitted tree over the @mib@ template itself.

Why it is deterministic

The shell pipeline this replaces – scripts/vendor.sh in pysnmp/mibs – ran mibdump once per vendor directory under GNU parallel, each invocation reading output/asn1 as a dependency source while every other invocation copied its own sources into that same directory on the way out. Four properties made its output depend on something other than its input, and this tool answers each of them (pysnmp/pysmi#182):

Nothing is written into a directory that is read. Every namespace is a source for the whole build, so a module resolves the same way whatever order the namespaces are processed in, and the driver refuses to start if an output directory is, contains or is contained by a source. Cross-namespace resolution was the sharp end of this: fifteen of the modules the old build could only resolve through output/asn1 live in a different vendor directory from the one importing them, so whether they resolved at all depended on which parallel job finished first.

Nothing is fetched. Sources are local directories and packages, and no borrowers are configured. The old build passed --mib-source=https://pysnmp.github.io/mibs/asn1/@mib@, resolving missing dependencies from its own last publish – so the corpus was not reproducible from the repository alone, and a bad publish perpetuated itself. Measured against the published corpus, dropping that source costs nothing: everything it serves that the build uses is already in the tree.

One error policy. Every output format compiles the same module set and keeps going past a defective module, and the run reports what failed. The old build set --ignore-errors on the jsondoc pass alone, and MibCompiler.compile() was all-or-nothing at the time, so a single defective module removed its entire namespace from notexts/ and texts/ while json/ kept it: 41% of the vendor corpus was published with JSON output and no Python. A corpus of MIBs nobody controls will always carry some that do not compile; what it owes is an inventory of them, which report.json is.

The index is decided by rule. pysmi.corpus.index ranks the modules defining an OID and takes the best one, term by term:

1. a module that still defines something current, over one whose objects are all obsolete;

2. tier – standard, then Internet-Draft, then vendor;

3. the module that registers an arc with its MODULE-IDENTITY, over one that merely names it with an OBJECT-IDENTITY;

4. the newest MODULE-IDENTITY revision, read as a date by normalise_revision – so a stamp that is not one, like HPR-MIB’s 970514000000Z, is refused rather than sorting above every date there will ever be;

5. the later RFC, then the module name, which settles the SMI root arcs and makes the rule total.

gen_index() is unchanged and still records the complete fact – every module defining a given OID. This is the projection of it a consumer that has to load exactly one module needs.

Two corpora from one source set

"publish": false is what lets one set of sources produce two corpora that differ in what they carry rather than in how they were built.

The full corpus carries the standard modules, because its consumers fetch them from it: splunk-connect-for-snmp resolves asn1/@mib@ against the published tree for every module it meets, standard ones included. This is what pysnmp/mibs serves today and what must keep working unchanged.

{"version": 1, "namespaces": [
  {"name": "standard", "source": "package:pysmi.mibs.asn1", "tier": "standard"},
  {"include": "src/vendor/*", "tier": "vendor"}]}

The compact corpus is the same source set with the standard namespace declared unpublished. It carries only what a runtime that already has the standard modules does not have – pysmi bundles 210 of them and its wheel ships their compiled form, so restating them costs size and says nothing new:

{"version": 1, "namespaces": [
  {"name": "standard", "source": "package:pysmi.mibs.asn1", "tier": "standard",
   "publish": false},
  {"include": "src/vendor/*", "tier": "vendor"}]}

Leaving the standard namespace out altogether is not the same thing: every vendor module that imports SNMPv2-SMI would fail to compile. The difference between the two manifests is what is carried, not what is resolved, which is why the compact corpus is a subset of the full one and not a second rendering of it – each module it carries is byte-identical to the same module in the full corpus, and its index covers exactly the modules it carries, so a runtime can merge it with whatever it already knows about the standard tree.

A build for the compact corpus normally asks for a subset of the artifacts too, since standard.txt and the frozen legacy index are compatibility surfaces of the published one:

$ mibcorpus --manifest=corpus-compact.json --output-directory=compact \
    --emit=asn1 --emit=notexts --emit=texts --emit=json \
    --emit=index-v2 --emit=report

Packaging is the caller’s: the driver writes a directory, and what a release does with it – an archive, a container image built from scratch for a runtime to mount – belongs to the repository publishing it, not to the compiler driver. What the driver owes such a consumer is that the directory is the same bytes every time, which is the property the tests above pin.

One driver, both corpora

pysmi’s own wheel is built with this driver. hatch_build.py compiles the 486 bundled ASN.1 modules into pysmi/mibs/pysnmp/ while the wheel is built, and it does that by running CorpusDriver over a corpus of one namespace – the bundle – rather than by keeping a compile loop of its own. A difference between how pysmi builds the base layer and how pysnmp/mibs builds on top of it would be a difference nobody chose.

The one thing the wheel configures differently is the stub list. A corpus published beside pysnmp does not restate what pysnmp already implements, so the default is mibdump’s: the base MIBs. The wheel is that base layer, and only three of those modules genuinely cannot be generated – the ones the generator emits an unconditional import from, which would become an import from itself. The other 11 generate perfectly well and consumers want them (pysnmp/pysmi#196), so the wheel passes the narrower list through the stubs argument.

Compatibility

--frozen-index is what keeps a published index answering what it has always answered. The legacy index is not regenerated: it replays the snapshot, drops rows naming a module the corpus no longer carries, and adds OIDs the snapshot never had. Corrections land in index-v2.csv.