MIB compiler

class pysmi.compiler.MibCompiler(parser: AbstractParser, codegen: AbstractCodeGen, writer: AbstractWriter, useBundledMibs: bool = True, preferConfiguredSources: bool = False, parseCache: AbstractParseCache | None = None)

Top-level, user-facing, composite MIB compiler object.

MibCompiler implements high-level MIB transformation processing logic. It executes its actions by calling the following specialized objects:

  • readers - to acquire ASN.1 MIB data

  • searchers - to see if transformed MIB already exists and no processing is necessary

  • parser - to parse ASN.1 MIB into AST

  • code generator - to perform actual MIB transformation

  • borrowers - to fetch pre-transformed MIB if transformation is impossible

  • writer - to store transformed MIB data

Required components must be passed to MibCompiler on instantiation. Those components are: parser, codegenerator and writer.

Optional components could be set or modified at later phases of MibCompiler life. Unlike singular, required components, optional one can be present in sequences to address many possible sources of data. They are readers, searchers and borrowers.

Creates an instance of MibCompiler class.

Parameters:

  • parser – ASN.1 MIB parser object

  • codegen – MIB transformation object

  • writer – transformed MIB storing object

Keyword Arguments:

  • useBundledMibs – register pysmi’s own bundled copy of the base MIBs (SNMPv2-SMI and friends) as a priority source, tried ahead of everything added through add_sources(). It is a source of its own rather than a last-resort fallback: where an add_sources() reader has one of those modules too, the newest MODULE-IDENTITY revision supplies it and this ordering only breaks the tie – compile() states the whole rule. Set to False to compile from add_sources() alone, so a misconfigured call fails loudly instead of silently succeeding from the bundled copy.

  • preferConfiguredSources – put add_sources() ahead of the bundled copy where the revisions do not decide – a module with no MODULE-IDENTITY to compare, or two copies carrying the same one. The newest revision still wins when there is one on every copy; this only settles what the bundle would otherwise settle by being asked first. Nearly half the bundled modules carry no MODULE-IDENTITY at all (SNMPv2-SMI and the other SMI modules among them), so for those this is the difference between the caller’s copy being used and the bundled one.

  • parseCache – where to keep parse trees between compile() calls, as an AbstractParseCache. A driver compiling many source sets on one compiler – see set_sources() – then parses the standard tree once instead of once per set. Defaults to InMemoryParseCache; FileParseCache survives process exit, for a build that is a shell loop rather than one process; NullParseCache turns caching off. A provider is used as passed and never resolved by name or from configuration, so the trust boundary is the caller’s own code – which matters, because a provider that stores trees outside this process reconstructs arbitrary Python objects when it reads them back.

addBorrowers(*borrowers: AbstractBorrower) → MibCompiler

Deprecated alias for add_borrowers().

addPrioritySources(*sources: AbstractReader) → MibCompiler

Deprecated alias for add_priority_sources().

addSearchers(*searchers: AbstractSearcher) → MibCompiler

Deprecated alias for add_searchers().

addSources(*sources: AbstractReader) → MibCompiler

Deprecated alias for add_sources().

add_borrowers(*borrowers: AbstractBorrower) → MibCompiler

Add more transformed MIBs repositories to borrow MIBs from.

Whenever MibCompiler.compile encounters MIB module which neither of the searchers can find or fetched ASN.1 MIB module can not be parsed (due to syntax errors), these borrowers objects will be invoked in order of their addition asking each if already transformed MIB can be fetched (borrowed).

Parameters:

borrowers – borrower object(s)

Returns:

reference to itself (can be used for call chaining)

add_priority_sources(*sources: AbstractReader) → MibCompiler

Add ASN.1 MIB source repositories to be asked ahead of the rest.

Every one of these is tried, for every MIB, before any add_sources() source, whatever order either was added in. Use it for a source that is more trustworthy than whatever the caller happens to have configured, such as pysmi’s own bundled base MIBs (see useBundledMibs on the constructor).

A distribution’s /usr/share/snmp/mibs routinely carries a base MIB frozen years ago, and taking that over a copy pinned to its RFC is almost never what anyone wanted. Overriding a bundled module is still possible – ship a newer MODULE-IDENTITY revision of it, pass preferConfiguredSources=True so add_sources() outranks the bundle wherever revisions do not decide, or pass useBundledMibs=False to drop the bundle entirely.

Parameters:

sources – reader object(s)

Returns:

reference to itself (can be used for call chaining)

add_searchers(*searchers: AbstractSearcher) → MibCompiler

Add more transformed MIBs repositories.

MibCompiler.compile will invoke each of configured searcher objects in order of their addition asking each if already transformed MIB module already exists and is more recent than specified.

Parameters:

searchers – searcher object(s)

Returns:

reference to itself (can be used for call chaining)

add_sources(*sources: AbstractReader) → MibCompiler

Add more ASN.1 MIB source repositories.

MibCompiler.compile will invoke each of configured source objects in order of their addition asking each to fetch MIB module specified by name. The first source that has a module supplies it, and every add_priority_sources() source is asked first, whatever order either was added in.

The one exception is a module pysmi bundles a copy of, where the newest MODULE-IDENTITY revision wins instead – when every copy found carries one – and this order only breaks the tie. Where it does fall to this order, preferConfiguredSources on the constructor puts these sources ahead of the bundled copy. compile() documents the whole rule.

Parameters:

sources – reader object(s)

Returns:

reference to itself (can be used for call chaining)

buildIndex(processedMibs: dict[str, MibStatus], **options: Any) → None

Deprecated alias for build_index().

build_index(processedMibs: dict[str, MibStatus], **options: Any) → None

Generate and store an index over the MIBs just compiled.

Parameters:

processedMibs – MIB module names mapped to their compilation results

Keyword Arguments:

  • dryRun – build the index but do not store it

  • ignoreErrors – log a failure to build the index instead of raising

Raises:

PySmiError – the index could not be built or stored, unless ignoreErrors is set.

bundledMibsPackage = 'pysmi.mibs.asn1'

Dotted package holding the bundled base MIB ASN.1 sources – read through PackageReader when useBundledMibs is set. See pysnmp/pysmi#113.

clearParseCache() → MibCompiler

Deprecated alias for clear_parse_cache().

clear_parse_cache() → MibCompiler

Drop every parse tree the configured cache holds.

Never needed for correctness – a key is derived from the text and its producer, so an entry is only ever reused for identical input. It is here for a caller that wants the space back at a known point.

Returns:

reference to itself (can be used for call chaining)

compile(*mibnames: str, **options: Any) → dict[str, MibStatus]

Transform requested and possibly referred MIBs.

The compile method should be invoked when MibCompiler object is operational meaning at least sources are specified.

Once called with a MIB module name, compile will:

  • fetch ASN.1 MIB module with given name by calling sources

  • make sure no such transformed MIB already exists (with searchers)

  • parse ASN.1 MIB text with parser

  • perform actual MIB transformation into target format with code generator

  • may attempt to borrow pre-transformed MIB through borrowers

  • write transformed MIB through writer

The above sequence will be performed for each MIB name given in mibnames and may be performed for all MIBs referred to from MIBs being processed.

Parameters:

  • mibnames – list of ASN.1 MIBs names

  • options – options that affect the way PySMI components work

Keyword Arguments:

strictSources – fail a MIB that more than one configured source has a different copy of, rather than taking one and reporting the other on MibStatus.shadowed.

Returns:

A dictionary of MIB module names processed (keys) and MibStatus class instances (values)

Note

Which copy of a module gets compiled, when more than one source has it:

  1. For a module pysmi bundles a copy of, the newest MODULE-IDENTITY LAST-UPDATED wins – provided every copy found carries one. An undated copy cannot be placed against a dated one, so a single one of those drops the module to rule 2 whatever the others carry.

  2. Otherwise – to break a tie between equal revisions, for a module with an undated copy, and for everything pysmi does not bundle – source order wins: every add_priority_sources() reader, then every add_sources() reader, each in the order it was added. preferConfiguredSources moves the bundled copy behind add_sources() for this rule only.

Rule 1 is deliberately confined to the bundled names. Those are pinned to an RFC or to IANA and re-checked against it, so two copies of one are the same specification at two revisions and the newer is simply better. Two copies of a vendor module are not that: they are a collision, or two firmware revisions, and which one was meant is what the caller’s source order says.

Rule 2 carries more than it looks like it does: 13 of the 27 bundled modules – SNMPv2-SMI, SNMPv2-TC, SNMPv2-CONF and the other SMI and RFC-numbered ones – have no MODULE-IDENTITY at all, so for those rule 1 can never fire and the bundled copy is used unless preferConfiguredSources or useBundledMibs=False says otherwise.

MibStatus.path names the file a module was compiled from, MibStatus.shadowed the copies passed over, and MibStatus.precedence which of these rules chose between them – so a build can record what it resolved to, why, and reproduce it later.

listMibs(includeBundled: bool = False) → list[str]

Deprecated alias for list_mibs().

list_mibs(includeBundled: bool = False) → list[str]

Every module name the configured sources can be asked to enumerate.

This is what turns “compile this collection” into a list of modules without anyone having to write the list down. A source that cannot be listed – a web server answering a @mib@ URL template – reports nothing, so what comes back is the modules held locally.

The bundled base MIBs are left out by default. They are a resolution source, supplying whatever a compiled module imports; a caller asking what to build normally means its own collection, not pysmi’s copy of the standard MIBs on top of it.

Keyword Arguments:

includeBundled – also enumerate the bundled base MIBs.

Returns:

Module names in source order, each appearing once.

prune(**options: Any) → dict[str, MibStatus]

Remove previously stored output whose source MIB no longer exists.

compile only ever acts on the MIBs it is asked for, so output for a module that has since been removed from every configured source lingers in the destination forever. prune closes that gap: it asks the writer what it currently holds (list_data(), which reports nothing for a writer that cannot enumerate its own output, e.g. CallbackWriter), and for each name tries every configured source in turn. A name none of them have any more is removed.

Only output carrying this package’s own “Produced by” marker is ever considered – a file the writer holds that this tool did not generate is left alone, whatever else is true of it.

Every source has clear_cache() called on it first, so a reader already warmed up by an earlier compile in this same run reports what exists right now rather than what existed when it was first asked.

Keyword Arguments:

  • dryRun – report what would be removed without removing anything

  • ignoreErrors – keep going after a source or writer error instead of raising

Returns:

A dictionary of MIB module names the writer held (keys) and MibStatus instances (values) – pruned if removed, untouched if a source still has it, failed if removal itself failed.

Raises:

PySmiError – a source or the writer failed, unless ignoreErrors is set.

resolve(mibname: str) → MibResolution | None

Which copy of mibname the configured sources supply, without compiling it.

compile() answers this on the way past, on MibStatus.path, MibStatus.shadowed and MibStatus.precedence – but only for a module it went on to compile, and only after it has. A driver publishing the ASN.1 beside the compiled output has to answer it for every module it holds, including the ones that fail, and has to answer it the same way the compile will. Asking here is what makes the two agree by construction rather than by inspection.

The rule is compile()’s own, applied by the same code: the newest MODULE-IDENTITY revision wins, source order breaks the tie.

Parameters:

mibname – MIB module name

Returns:

What the sources hold for that name, or None when none of them has it.

setSources(*sources: AbstractReader) → MibCompiler

Deprecated alias for set_sources().

set_sources(*sources: AbstractReader) → MibCompiler

Replace the ASN.1 sources, keeping everything else about this compiler.

For a driver compiling many source sets in turn – a corpus build over several hundred vendor namespaces, each with its own directory and each importing the same standard modules. Building a fresh compiler per set re-parses that standard tree once per set, because the parse cache lives in compile() and dies with the call; swapping the sources on one compiler keeps it.

Only add_sources() readers are replaced. add_priority_sources() ones, the bundled base MIBs among them, are left in place – they are the part that does not vary between namespaces, and re-registering them per set is what this exists to avoid.

Swapping sources cannot make a stale answer reachable: the parse cache is keyed by the digest of the text parsed, never by module name, so a namespace carrying a different module under a name another namespace used is different bytes and a different key. What a swap does change is which sources are asked, and that takes effect immediately – a module only the previous set had stops resolving.

Parameters:

sources – reader object(s) to use from now on

Returns:

reference to itself (can be used for call chaining)

Compiling many source sets

A corpus build compiles a few hundred vendor namespaces, each with its own source directory, and every one of them imports the same handful of standard modules.

The parse cache belongs to the compiler, so it is kept across compile() calls and the standard tree is parsed once. What loses it is building a new compiler per namespace – and a shell loop, which starts a fresh process each time, can do nothing else. That is the shape the standard tree ends up parsed once per namespace in.

set_sources() replaces the configured sources on a compiler that is already built, so one compiler can be driven across every namespace instead:

from pysmi.codegen import JsonCodeGen
from pysmi.compiler import MibCompiler
from pysmi.parser import SmiV1CompatParser
from pysmi.reader import FileReader
from pysmi.writer import CallbackWriter

documents = {}

def store(mibname, data, cbCtx):
    documents[mibname] = data

compiler = MibCompiler(
    SmiV1CompatParser(), JsonCodeGen(), CallbackWriter(store)
)

for namespace in namespaces:
    compiler.set_sources(FileReader(namespace.path))
    compiler.compile(*namespace.modules)

Only add_sources() readers are replaced. The add_priority_sources() ones – pysmi’s bundled base MIBs among them – stay registered, because they are the part that does not vary between namespaces.

Measured over eight namespaces against the bundled sources:

wall

parses

distinct

a fresh compiler each

0.85s

45

14

one compiler, sources swapped

0.45s

14

14

Every parse in the second row is of text not parsed before in that build. The saving grows with the namespace count, since the shared tree is a fixed cost paid once rather than once per namespace.

Why swapping sources cannot serve a stale answer

The cache is keyed by the digest of the text that was parsed, never by module name. Two namespaces carrying different modules under one name – ordinary in a vendor corpus, where a MIB name recurs across trees at different revisions – are different bytes and therefore different keys, so neither can be served the other’s tree.

What a swap does change is which sources are asked, and that takes effect at once: a module only the previous source set had stops resolving.

Choosing where the trees are kept

The cache is a component, passed to the constructor. The Parse caches section of the library reference lists the ones pysmi ships and the interface a provider implements.

InMemoryParseCache is the default: bounded, least-recently-used evicted, so the shared modules stay resident while per-namespace ones do not accumulate. It is the right answer whenever the whole build runs in one process.

FileParseCache keeps trees in a directory, so they outlive the process that built them. That is what a build driven as a shell loop needs – a fresh interpreter per namespace has nothing in memory to reuse:

from pysmi.cache import FileParseCache

compiler = MibCompiler(
    SmiV1CompatParser(),
    JsonCodeGen(),
    CallbackWriter(store),          # as above
    parseCache=FileParseCache("/var/cache/pysmi"),
)

Reading that cache unpickles, which reconstructs arbitrary Python objects, so point it only at a directory your own build writes. The in-memory provider has no such boundary. A provider is always used as passed and never resolved by name, from an entry point or from configuration – the trust boundary is the calling code.

NullParseCache turns caching off, which is the behaviour of every release before it existed.

To write your own – Redis, memcached, a shared filesystem, whatever a build already runs – implement AbstractParseCache. No registration step exists because none is needed. Two things a provider does not have to do: invalidation, since the key already carries the text together with the identity of what would parse it – the pysmi version, the parser class, its grammar relaxations and its start symbol – so an entry written by another release, or by a different SMI dialect, is never read by this one; and defensive copying, since the compiler copies whatever it receives.

The dialect matters as much as the version here. parserFactory names every specialization it builds SmiParser, so all three shipped parsers share a class identity while accepting different grammars: a trailing comma in IMPORTS parses under SmiV1CompatParser and raises under SmiV2Parser.

clear_parse_cache() empties whichever provider is configured. It is never needed for correctness.