The *mibdump* tool
******************

The *mibdump.py* tool is a command-line frontend to the PySMI library.
This tool can be used for automatic downloading and transforming SNMP
MIB modules into various formats.

   $ mibdump.py --help
   Synopsis:
     SNMP SMI/MIB files conversion tool
   Documentation:
     http://snmplabs.com/pysmi
   Usage: mibdump.py [--help]
         [--version]
         [--quiet]
         [--debug=<all|borrower|codegen|compiler|grammar|lexer|parser|reader|searcher|writer>]
         [--mib-source=<URI>]
         [--mib-searcher=<PATH|PACKAGE>]
         [--mib-stub=<MIB-NAME>]
         [--mib-borrower=<PATH>]
         [--destination-format=<FORMAT>]
         [--destination-directory=<DIRECTORY>]
         [--cache-directory=<DIRECTORY>]
         [--disable-fuzzy-source]
         [--no-dependencies]
         [--no-python-compile]
         [--python-optimization-level]
         [--ignore-errors]
         [--build-index]
         [--rebuild]
         [--dry-run]
         [--no-mib-writes]
         [--generate-mib-texts]
         [--keep-texts-layout]
         <MIB-NAME> [MIB-NAME [...]]]
   Where:
       URI      - file, zip, http, https, ftp, sftp schemes are supported.
                  Use @mib@ placeholder token in URI to refer directly to
                  the required MIB module when source does not support
                  directory listing (e.g. HTTP).
       FORMAT   - pysnmp, json, null

When JSON destination format is requested, for each MIB module
*mibdump.py* will produce a JSON document containing all MIB objects.
For example, IF-MIB module in JSON form would look like:

   {
      "ifMIB": {
          "name": "ifMIB",
          "oid": "1.3.6.1.2.1.31",
          "class": "moduleidentity",
          "revisions": [
            "2007-02-15 00:00",
            "1996-02-28 21:55",
            "1993-11-08 21:55"
          ]
        },

      ...
      "ifTestTable": {
        "name": "ifTestTable",
        "oid": "1.3.6.1.2.1.31.1.3",
        "class": "objecttype",
        "maxaccess": "not-accessible"
      },
      "ifTestEntry": {
        "name": "ifTestEntry",
        "oid": "1.3.6.1.2.1.31.1.3.1",
        "class": "objecttype",
        "maxaccess": "not-accessible",
        "augmention": {
          "name": "ifTestEntry",
          "module": "IF-MIB",
          "object": "ifEntry"
        }
      },
      "ifTestId": {
        "name": "ifTestId",
        "oid": "1.3.6.1.2.1.31.1.3.1.1",
        "class": "objecttype",
        "syntax": {
          "type": "TestAndIncr",
          "class": "type"
        },
        "maxaccess": "read-write"
      },
      ...
   }

In general, JSON MIB captures all aspects of original (ASN.1) MIB
contents and layout. The snippet above is just an example, here is the
complete IF-MIB.json file.


Specifying MIB source
=====================

The --mib-source option can be given multiple times. Each instance of
--mib-source must specify a URL where ASN.1 MIB modules should be
looked up and downloaded from. At this moment three MIB sourcing
methods are supported:

* Local files. This could be a top-level directory where MIB files
  are located. Subdirectories will be automatically traversed as well.
  Example: file:///usr/share/snmp

* ZIP archives containing MIB files. Subdirectories and embedded ZIP
  archives will be automatically traversed. Example: zip://mymibs.zip

* HTTP/HTTPS. A fully specified URL where MIB module name is
  specified by a @mib@ placeholder. When specific MIB is looked up,
  PySMI will replace that placeholder with MIB module name it is
  looking for. Example: http://mibs.snmplabs.com/asn1/@mib@

* SFTP/FTP. A fully specified URL including FTP username and
  password. MIB module name is specified by a @mib@ placeholder. When
  specific MIB is looked up, PySMI will replace that placeholder with
  MIB module name it is looking for. Example:
  http://mibs.snmplabs.com/asn1/@mib@

When trying to fetch a MIB module, the *mibdump.py* tool will try each
of configured --mib-source transports in order of specification till
first successful hit.

By default *mibdump.py* will search:

* file:///usr/share/snmp

* http://mibs.snmplabs.com/asn1/@mib@

Once another --mib-source option is given, those defaults will not be
used and should be manually given to *mibdump.py* if needed.


Fuzzying MIB module names
=========================

There is no single convention on how MIB module files should be named.
By default *mibdump.py* will try a handful of guesses when trying to
find a file containing specific MIB module. It will try upper and
lower cases, a file named after MIB module, try adding different
extensions to a file (.mib, .my etc), try adding/cutting the '-MIB'
part of the file name. If nothing matches, *mibdump.py* will consider
that probed --mib-source does not contain MIB module it is looking
for.

There is a small chance, though, that fuzzy natching may result in
getting a wrong MIB. If that happens, you can disable the above
fuzzyness by giving *mibdump.py* the --disable-fuzzy-source flag.


Avoiding excessive transformation
=================================

It well may happen that many MIB modules refer to a common single MIB
module. In that case *mibdump.py* may transform it many times unless
you tell *mibdump.py* where to search for already transformed MIBs.
That place could of course be a directory where *mibdump.py* writes
its transforms into and/or some other local locations.

The --mib-searcher option specifies either local directory or
importable Python package (applicable to pysnmp transformation)
containing transformed MIB modules. Multiple --mib-searcher options
could be given, *mibdump.py* will use each of them in order of
specification till first hit.

If no transformed MIB module is found, *mibdump.py* will go on running
its full transformation cycle.

By default *mibdump.py* will use:

* --mib-searcher=$HOME/.pysnmp/mibs

* --mib-searcher=pysnmp_mibs

Once another --mib-searcher option is given, those defaults will not
be used and should be manually given to *mibdump.py* if needed.


Blacklisting MIBs
=================

Some MIBs may not be automatically transformed into another form and
therefore must be explicitly excluded from processing. Such MIBs are
normally manually implemented for each target MIB format. Examples
include MIBs containing base SMI types or ASN.1 MACRO definitions
(SNMPv2-SMI, SNMPV2-TC), initially compiled but later manually
modified MIBs and others.

Default list of blacklisted MIBs for pysnmp transformation target is:
RFC-1212, RFC-1215, RFC1065-SMI, RFC1155-SMI, RFC1158-MIB,
RFC1213-MIB, SNMP-FRAMEWORK-MIB, SNMP-TARGET-MIB, SNMPv2-CONF,
SNMPv2-SMI, SNMPv2-TC, SNMPv2-TM, TRANSPORT-ADDRESS-MIB.

If you need to modify this list use the --mib-stub option.


Dealing with broken MIBs
========================

Curiously enough, some MIBs coming from quite prominent vendors appear
syntactically incorrect. That leads to MIB compilers fail on such
MIBs. While many MIB compiler implementations (PySMI included)
introduce workarounds and grammar relaxations allowing slightly broken
MIBs to compile, however severely broken MIBs can't be reliably
compiled.

As another workaround PySMI offers the *borrow* feature. It allows
PySMI to fetch already transformed MIBs even if corresponding ASN.1
MIB can't be found or parsed.

Default source of pre-compiled MIBs for pysnmp target is:

* http://mibs.snmplabs.com/pysnmp/fulltexts/@mib@

* http://mibs.snmplabs.com/pysnmp/notexts/@mib@

If you wish to modify this default list use one or more --mib-borrower
options.


Choosing target transformation
==============================

PySMI design allows many transformation formats to be supported in
form of specialized code generation components. At the moment PySMI
can produce MIBs in form of pysnmp classes and JSON documents.

JSON document schema is chosen to preserve as much of MIB information
as possible. There's no established JSON schema known to the authors.


Setting destination directory
=============================

By default *mibdump.py* writes pysnmp MIBs into:

* $HOME/.pysnmp/mibs  (on UNIX)

* @HOME@PySNMP ConfigurationMIBs (on Windows)

and JSON files in current working directory.

Use --destination-directory option to change default output diretory.


Performing unconditional transformation
=======================================

By default PySMI will avoid creating new transformations if fresh
enough versions already exist. By using --rebuild option you could
trick PySMI doing requested transformation for all given MIB modules.


Ignoring transformation errors
==============================

By default PySMI will stop on first fatal error occurred during
transformations of a series of MIBs. If you wish PySMI to ignore fatal
errors and therefore skipping failed MIB, use the --ignore-errors
option.

Keep in mind that skipping transformation of MIBs that are imported by
other MIBs might make dependant MIBs inconsistent for use.


Skipping dependencies
=====================

Most MIBs rely on other MIBs for their operations. This is indicated
by the IMPORT statement in ASN.1 language. PySMI attempts to transform
all MIBs IMPORT'ed by MIB being transformed. That is done in
recurrsive manner.

By using --no-dependencies flag you can tell PySMI not to transform
any MIBs other than those explicitly requested to be transformed.

Keep in mind that skipping dependencies may make the whole set of
transformed MIBs inconsistent.


Generating MIB texts
====================

Most MIBs are very verbose. They contain many human-oriented
descriptions and clarifications written in plain English. Those texts
may be useful for MIB browser applications (to display those texts to
human operator) but might not make any sense in other applications.

To save space and CPU time, PySMI does not by default include those
texts into transformed MIBs. However this can be reverted by adding
--generate-mib-texts option.

When MIB texts are generated, whitespaces and new lines are stripped
by default. Sometimes that breaks down ASCII art should it occur in
MIB texts. To preserve original text formatting, --keep-texts-layout
option may be used.


Building MIB indices
====================

If --build-index option is given, depending on the destination format
chosen, the *mibdump.py* tool may create new (or update existing)
document containing MIB information in a form that is convenient for
querying cornerstone properties of MIB files.

For example, building JSON index for IP-MIB.json, TCP-MIB.json and
UDP-MIB.json MIB modules would emit something like this:

   {
      "compliance": {
         "1.3.6.1.2.1.48.2.1.1": [
           "IP-MIB"
         ],
         "1.3.6.1.2.1.49.2.1.1": [
           "TCP-MIB"
         ],
         "1.3.6.1.2.1.50.2.1.1": [
           "UDP-MIB"
         ]
      },
      "identity": {
          "1.3.6.1.2.1.48": [
            "IP-MIB"
          ],
          "1.3.6.1.2.1.49": [
            "TCP-MIB"
          ],
          "1.3.6.1.2.1.50": [
            "UDP-MIB"
          ]
      },
      "oids": {
          "1.3.6.1.2.1.4": [
            "IP-MIB"
          ],
          "1.3.6.1.2.1.5": [
            "IP-MIB"
          ],
          "1.3.6.1.2.1.6": [
            "TCP-MIB"
          ],
          "1.3.6.1.2.1.7": [
            "UDP-MIB"
          ],
          "1.3.6.1.2.1.49": [
            "TCP-MIB"
          ],
          "1.3.6.1.2.1.50": [
            "UDP-MIB"
          ]
      }
   }

With this example, *compliance* and *identity* keys point to *MODULE-
COMPLIANCE* and *MODULE-IDENTITY* MIB objects, *oids* list top-level
OIDs branches defined in MIB modules. Full index build over thousands
of MIBs could be seen here.


Minor speedups
==============

There are a few options that may improve PySMI performance.

The --cache-directory option may be used to point to a temporary
writable directory where PySMI parser (e.g. Ply) would store its
lookup tables.

By default PySMI performing transformation into pysnmp format will
also pre-compile Python source into interpreter bytecode. That takes
some time and space. If you wish not to cache Python bytecode or to do
that later, use the --no-python-compile option.
