PoolManager¶
A pool manager is an abstraction for a collection of ConnectionPools.
If you need to make requests to multiple hosts, then you can use a
PoolManager, which takes care of maintaining your pools
so you don’t have to.
>>> from urllib3 import PoolManager
>>> manager = PoolManager(10)
>>> r = manager.request('GET', 'http://example.com')
>>> r.headers['server']
'ECS (iad/182A)'
>>> r = manager.request('GET', 'http://httpbin.org/')
>>> r.headers['server']
'gunicorn/18.0'
>>> r = manager.request('POST', 'http://httpbin.org/headers')
>>> r = manager.request('HEAD', 'http://httpbin.org/cookies')
>>> len(manager.pools)
2
>>> conn = manager.connection_from_host('httpbin.org')
>>> conn.num_requests
3
A PoolManager will create a new ConnectionPool
when no ConnectionPools exist with a matching pool key.
The pool key is derived using the requested URL and the current values
of the connection_pool_kw instance variable on PoolManager.
The keys in connection_pool_kw used when deriving the key are
configurable. For example, by default the my_field key is not
considered.
>>> from urllib3.poolmanager import PoolManager
>>> manager = PoolManager(10, my_field='wheat')
>>> manager.connection_from_url('http://example.com')
>>> manager.connection_pool_kw['my_field'] = 'barley'
>>> manager.connection_from_url('http://example.com')
>>> len(manager.pools)
1
To make the pool manager create new pools when the value of
my_field changes, you can define a custom pool key and alter
the key_fn_by_scheme instance variable on PoolManager.
>>> import functools
>>> from collections import namedtuple
>>> from urllib3.poolmanager import PoolManager, HTTPPoolKey
>>> from urllib3.poolmanager import default_key_normalizer as normalizer
>>> CustomKey = namedtuple('CustomKey', HTTPPoolKey._fields + ('my_field',))
>>> manager = PoolManager(10, my_field='wheat')
>>> manager.key_fn_by_scheme['http'] = functools.partial(normalizer, CustomKey)
>>> manager.connection_from_url('http://example.com')
>>> manager.connection_pool_kw['my_field'] = 'barley'
>>> manager.connection_from_url('http://example.com')
>>> len(manager.pools)
2
The API of a PoolManager object is similar to that of a
ConnectionPool, so they can be passed around interchangeably.
The PoolManager uses a Least Recently Used (LRU) policy for discarding old
pools. That is, if you set the PoolManager num_pools to 10, then after
making requests to 11 or more different hosts, the least recently used pools
will be cleaned up eventually.
Cleanup of stale pools does not happen immediately but can be forced when used as a context manager.
>>> from urllib3 import PoolManager
>>> with PoolManager(10) as manager:
... r = manager.request('GET', 'http://example.com')
... r = manager.request('GET', 'http://httpbin.org/')
... len(manager.pools)
...
2
>>> len(manager.pools)
0
You can read more about the implementation and the various adjustable variables
within RecentlyUsedContainer.
API¶
- class
urllib3.poolmanager.PoolManager(num_pools=10, headers=None, **connection_pool_kw)¶Allows for arbitrary requests while transparently keeping track of necessary connection pools for you.
Parameters:
- num_pools – Number of connection pools to cache before discarding the least recently used pool.
- headers – Headers to include with all requests, unless other headers are given explicitly.
- **connection_pool_kw – Additional parameters are used to create fresh
urllib3.connectionpool.ConnectionPoolinstances.Example:
>>> manager = PoolManager(num_pools=2) >>> r = manager.request('GET', 'http://google.com/') >>> r = manager.request('GET', 'http://google.com/mail') >>> r = manager.request('GET', 'http://yahoo.com/') >>> len(manager.pools) 2
clear()¶Empty our store of pools and direct them all to close.
This will not affect in-flight connections, but they will not be re-used after completion.
connection_from_context(request_context)¶Get a
ConnectionPoolbased on the request context.
request_contextmust at least contain theschemekey and its value must be a key inkey_fn_by_schemeinstance variable.
connection_from_host(host, port=None, scheme='http')¶Get a
ConnectionPoolbased on the host, port, and scheme.If
portisn’t given, it will be derived from theschemeusingurllib3.connectionpool.port_by_scheme.
connection_from_pool_key(pool_key)¶Get a
ConnectionPoolbased on the provided pool key.
pool_keyshould be a namedtuple that only contains immutable objects. At a minimum it must have thescheme,host, andportfields.
connection_from_url(url)¶Similar to
urllib3.connectionpool.connection_from_url()but doesn’t pass any additional parameters to theurllib3.connectionpool.ConnectionPoolconstructor.Additional parameters are taken from the
PoolManagerconstructor.
request(method, url, fields=None, headers=None, **urlopen_kw)¶Make a request using
urlopen()with the appropriate encoding offieldsbased on themethodused.This is a convenience method that requires the least amount of manual effort. It can be used in most situations, while still having the option to drop down to more specific methods when necessary, such as
request_encode_url(),request_encode_body(), or even the lowest levelurlopen().
request_encode_body(method, url, fields=None, headers=None, encode_multipart=True, multipart_boundary=None, **urlopen_kw)¶Make a request using
urlopen()with thefieldsencoded in the body. This is useful for request methods like POST, PUT, PATCH, etc.When
encode_multipart=True(default), thenurllib3.filepost.encode_multipart_formdata()is used to encode the payload with the appropriate content type. Otherwiseurllib.urlencode()is used with the ‘application/x-www-form-urlencoded’ content type.Multipart encoding must be used when posting files, and it’s reasonably safe to use it in other times too. However, it may break request signing, such as with OAuth.
Supports an optional
fieldsparameter of key/value strings AND key/filetuple. A filetuple is a (filename, data, MIME type) tuple where the MIME type is optional. For example:fields = { 'foo': 'bar', 'fakefile': ('foofile.txt', 'contents of foofile'), 'realfile': ('barfile.txt', open('realfile').read()), 'typedfile': ('bazfile.bin', open('bazfile').read(), 'image/jpeg'), 'nonamefile': 'contents of nonamefile field', }When uploading a file, providing a filename (the first parameter of the tuple) is optional but recommended to best mimick behavior of browsers.
Note that if
headersare supplied, the ‘Content-Type’ header will be overwritten because it depends on the dynamic random boundary string which is used to compose the body of the request. The random boundary string can be explicitly set with themultipart_boundaryparameter.
request_encode_url(method, url, fields=None, headers=None, **urlopen_kw)¶Make a request using
urlopen()with thefieldsencoded in the url. This is useful for request methods like GET, HEAD, DELETE, etc.
urlopen(method, url, redirect=True, **kw)¶Same as
urllib3.connectionpool.HTTPConnectionPool.urlopen()with custom cross-host redirect logic and only sends the request-uri portion of theurl.The given
urlparameter must be absolute, such that an appropriateurllib3.connectionpool.ConnectionPoolcan be chosen for it.
- class
urllib3.poolmanager.BasePoolKey(scheme, host, port)¶
count(value) → integer -- return number of occurrences of value¶
host¶Alias for field number 1
index(value[, start[, stop]]) → integer -- return first index of value.¶Raises ValueError if the value is not present.
port¶Alias for field number 2
scheme¶Alias for field number 0
- class
urllib3.poolmanager.HTTPPoolKey(scheme, host, port, timeout, retries, strict, block, source_address)¶
block¶Alias for field number 6
count(value) → integer -- return number of occurrences of value¶
host¶Alias for field number 1
index(value[, start[, stop]]) → integer -- return first index of value.¶Raises ValueError if the value is not present.
port¶Alias for field number 2
retries¶Alias for field number 4
scheme¶Alias for field number 0
source_address¶Alias for field number 7
strict¶Alias for field number 5
timeout¶Alias for field number 3
- class
urllib3.poolmanager.HTTPSPoolKey(scheme, host, port, timeout, retries, strict, block, source_address, key_file, cert_file, cert_reqs, ca_certs, ssl_version, ca_cert_dir)¶
block¶Alias for field number 6
ca_cert_dir¶Alias for field number 13
ca_certs¶Alias for field number 11
cert_file¶Alias for field number 9
cert_reqs¶Alias for field number 10
count(value) → integer -- return number of occurrences of value¶
host¶Alias for field number 1
index(value[, start[, stop]]) → integer -- return first index of value.¶Raises ValueError if the value is not present.
key_file¶Alias for field number 8
port¶Alias for field number 2
retries¶Alias for field number 4
scheme¶Alias for field number 0
source_address¶Alias for field number 7
ssl_version¶Alias for field number 12
strict¶Alias for field number 5
timeout¶Alias for field number 3
ProxyManager¶
ProxyManager is an HTTP proxy-aware subclass of PoolManager.
It produces a single
HTTPConnectionPool instance for all HTTP
connections and individual per-server:port
HTTPSConnectionPool instances for tunnelled
HTTPS connections.
Example using proxy authentication:
>>> headers = urllib3.make_headers(proxy_basic_auth='myusername:mypassword')
>>> proxy = urllib3.ProxyManager('http://localhost:3128', proxy_headers=headers)
>>> r = proxy.request('GET', 'http://example.com/')
>>> r.status
200
API¶
- class
urllib3.poolmanager.ProxyManager(proxy_url, num_pools=10, headers=None, proxy_headers=None, **connection_pool_kw)¶Behaves just like
PoolManager, but sends all requests through the defined proxy, using the CONNECT method for HTTPS URLs.
Parameters:
- proxy_url – The URL of the proxy to be used.
- proxy_headers – A dictionary contaning headers that will be sent to the proxy. In case of HTTP they are being sent with each request, while in the HTTPS/CONNECT case they are sent only once. Could be used for proxy authentication.
- Example:
>>> proxy = urllib3.ProxyManager('http://localhost:3128/') >>> r1 = proxy.request('GET', 'http://google.com/') >>> r2 = proxy.request('GET', 'http://httpbin.org/') >>> len(proxy.pools) 1 >>> r3 = proxy.request('GET', 'https://httpbin.org/') >>> r4 = proxy.request('GET', 'https://twitter.com/') >>> len(proxy.pools) 3