XML Catalogs

When loading an external entity for a document, e.g. a DTD, the parser is normally configured to prevent network access (see the no_network parser option). Instead, it will try to load the entity from their local file system path or, in the most common case that the entity uses a network URL as reference, from a local XML catalog.

XML catalogs are the preferred and agreed-on mechanism to load external entities from XML processors. Most tools will use them, so it is worth configuring them properly on a system. Many Linux installations use them by default, but on other systems they may need to get enabled manually. The libxml2 site has some documentation on how to set up XML catalogs

URI Resolvers

Here is an example of a custom resolver:

>>> from lxml import etree

>>> class DTDResolver(etree.Resolver):
...     def resolve(self, url, id, context):
...         print("Resolving URL '%s'" % url)
...         return self.resolve_string(
...             '<!ENTITY myentity "[resolved text: %s]">' % url, context)

This defines a resolver that always returns a dynamically generated DTD fragment defining an entity. The url argument passes the system URL of the requested document, the id argument is the public ID. Note that any of these may be None. The context object is not normally used by client code.

Resolving is based on the methods of the Resolver object that build internal representations of the result document. The following methods exist:

• resolve_string takes a parsable string as result document
• resolve_filename takes a filename
• resolve_file takes an open file-like object that has at least a read() method
• resolve_empty resolves into an empty document

The resolve() method may choose to return None, in which case the next registered resolver (or the default resolver) is consulted. Resolving always terminates if resolve() returns the result of any of the above resolve_*() methods.

Resolvers are registered local to a parser:

>>> parser = etree.XMLParser(load_dtd=True)
>>> parser.resolvers.add( DTDResolver() )

Note that we instantiate a parser that loads the DTD. This is not done by the default parser, which does no validation. When we use this parser to parse a document that requires resolving a URL, it will call our custom resolver:

>>> xml = '<!DOCTYPE doc SYSTEM "MissingDTD.dtd"><doc>&myentity;</doc>'
>>> tree = etree.parse(StringIO(xml), parser)
Resolving URL 'MissingDTD.dtd'
>>> root = tree.getroot()
>>> print(root.text)
[resolved text: MissingDTD.dtd]

The entity in the document was correctly resolved by the generated DTD fragment.

Document loading in context

XML documents memorise their initial parser (and its resolvers) during their life-time. This means that a lookup process related to a document will use the resolvers of the document's parser. We can demonstrate this with a resolver that only responds to a specific prefix:

>>> class PrefixResolver(etree.Resolver):
...     def __init__(self, prefix):
...         self.prefix = prefix
...         self.result_xml = '''\
...              <xsl:stylesheet
...                     xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
...                <test xmlns="testNS">%s-TEST</test>
...              </xsl:stylesheet>
...              ''' % prefix
...     def resolve(self, url, pubid, context):
...         if url.startswith(self.prefix):
...             print("Resolved url %s as prefix %s" % (url, self.prefix))
...             return self.resolve_string(self.result_xml, context)

We demonstrate this in XSLT and use the following stylesheet as an example:

>>> xml_text = """\
... <xsl:stylesheet version="1.0"
...    xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
...   <xsl:include href="honk:test"/>
...   <xsl:template match="/">
...     <test>
...       <xsl:value-of select="document('hoi:test')/*/*/text()"/>
...     </test>
...   </xsl:template>
... </xsl:stylesheet>
... """

Note that it needs to resolve two URIs: honk:test when compiling the XSLT document (i.e. when resolving xsl:import and xsl:include elements) and hoi:test at transformation time, when calls to the document function are resolved. If we now register different resolvers with two different parsers, we can parse our document twice in different resolver contexts:

>>> hoi_parser = etree.XMLParser()
>>> normal_doc = etree.parse(StringIO(xml_text), hoi_parser)

>>> hoi_parser.resolvers.add( PrefixResolver("hoi") )
>>> hoi_doc = etree.parse(StringIO(xml_text), hoi_parser)

>>> honk_parser = etree.XMLParser()
>>> honk_parser.resolvers.add( PrefixResolver("honk") )
>>> honk_doc = etree.parse(StringIO(xml_text), honk_parser)

These contexts are important for the further behaviour of the documents. They memorise their original parser so that the correct set of resolvers is used in subsequent lookups. To compile the stylesheet, XSLT must resolve the honk:test URI in the xsl:include element. The hoi resolver cannot do that:

>>> transform = etree.XSLT(normal_doc)
Traceback (most recent call last):
  ...
lxml.etree.XSLTParseError: Cannot resolve URI honk:test

>>> transform = etree.XSLT(hoi_doc)
Traceback (most recent call last):
  ...
lxml.etree.XSLTParseError: Cannot resolve URI honk:test

However, if we use the honk resolver associated with the respective document, everything works fine:

>>> transform = etree.XSLT(honk_doc)
Resolved url honk:test as prefix honk

Running the transform accesses the same parser context again, but since it now needs to resolve the hoi URI in the call to the document function, its honk resolver will fail to do so:

>>> result = transform(normal_doc)
Traceback (most recent call last):
  ...
lxml.etree.XSLTApplyError: Cannot resolve URI hoi:test

>>> result = transform(hoi_doc)
Traceback (most recent call last):
  ...
lxml.etree.XSLTApplyError: Cannot resolve URI hoi:test

>>> result = transform(honk_doc)
Traceback (most recent call last):
  ...
lxml.etree.XSLTApplyError: Cannot resolve URI hoi:test

This can only be solved by adding a hoi resolver to the original parser:

>>> honk_parser.resolvers.add( PrefixResolver("hoi") )
>>> result = transform(honk_doc)
Resolved url hoi:test as prefix hoi
>>> print(str(result)[:-1])
<?xml version="1.0"?>
<test>hoi-TEST</test>

We can see that the hoi resolver was called to generate a document that was then inserted into the result document by the XSLT transformation. Note that this is completely independent of the XML file you transform, as the URI is resolved from within the stylesheet context:

>>> result = transform(normal_doc)
Resolved url hoi:test as prefix hoi
>>> print(str(result)[:-1])
<?xml version="1.0"?>
<test>hoi-TEST</test>

It may be seen as a matter of taste what resolvers the generated document inherits. For XSLT, the output document inherits the resolvers of the input document and not those of the stylesheet. Therefore, the last result does not inherit any resolvers at all.

I/O access control in XSLT

By default, XSLT supports all extension functions from libxslt and libexslt as well as Python regular expressions through EXSLT. Some extensions enable style sheets to read and write files on the local file system.

XSLT has a mechanism to control the access to certain I/O operations during the transformation process. This is most interesting where XSL scripts come from potentially insecure sources and must be prevented from modifying the local file system. Note, however, that there is no way to keep them from eating up your precious CPU time, so this should not stop you from thinking about what XSLT you execute.

Access control is configured using the XSLTAccessControl class. It can be called with a number of keyword arguments that allow or deny specific operations:

>>> transform = etree.XSLT(honk_doc)
Resolved url honk:test as prefix honk
>>> result = transform(normal_doc)
Resolved url hoi:test as prefix hoi

>>> ac = etree.XSLTAccessControl(read_network=False)
>>> transform = etree.XSLT(honk_doc, access_control=ac)
Resolved url honk:test as prefix honk
>>> result = transform(normal_doc)
Traceback (most recent call last):
  ...
lxml.etree.XSLTApplyError: xsltLoadDocument: read rights for hoi:test denied

There are a few things to keep in mind:

• XSL parsing (xsl:import, etc.) is not affected by this mechanism
• read_file=False does not imply write_file=False, all controls are independent.
• read_file only applies to files in the file system. Any other scheme for URLs is controlled by the *_network keywords.
• If you need more fine-grained control than switching access on and off, you should consider writing a custom document loader that returns empty documents or raises exceptions if access is denied.