Redis连接池
Redis 连接池
ConnectionPool的member function包括:
- from_url 从redis url 初始化连接池
- init 初始化连接池
- repr 对象表达
- reset 重置内部成员变量
- _checkpid
- get_connection 获取一个连接
- get_encoder 入参/出参 编解码
- make_connection 创建连接
- release 释放一个连接
- owns_connection 是否拥有一个连接
- disconnect 断开空闲连接或者断开所有连接
ConnectionPool 的成员变量:
self.connection_class = connection_class
self.connection_kwargs = connection_kwargs
self.max_connections = max_connections
self._fork_lock = threading.Lock()
self._created_connections = 0
self._available_connections = []
self._in_use_connections = set()
成员变量解释
connection_class
在connection.py中有三个Connection类型,分别是:
class Connection(object):
...
class SSLConnection(object):
...
class UnixDomainSocketConnection(Connection):
...
其中,Connection是最基本的连接对象 SSLConnection 继承了Connection,并且重写了_connect方法。Redis 6.0 开始支持SSL/TLS加密数据传输,如果Redis Server需要进行加密数据传输,就需要通过SSLConnection建立连接和传输数据。 对应的redis-cli连接方式如下:
./src/redis-cli --tls \
--cert ./tests/tls/redis.crt \
--key ./tests/tls/redis.key \
--cacert ./tests/tls/ca.crt
UnixDomainSocketConnection 同样继承自Connection,并且重写了__init__、_connect、repr_pieces、_error_message方法。 当使用unix socket的方式启动Redis Server时,需要通过这种类型的连接。
关于unixSocket vs tcp socket的启动方式,哪一个比较快,可以参考链接
connection_kwargs
建立连接的扩展参数取决于connection_class,以Connection 类为例,connection_kwargs的可选keys为:
host='localhost', # socket 三要素之IP地址
port=6379, # socket 三要素之端口号
db=0, # Redis 默认支持16个DB,但是不建议修改,阿里云的云Redis默认支持256个DB,每个DB没有单独的内存占用量的限制,DB可以使用的内存容量受Redis实例的总内存的限制,可以通过 select 10,变更数据库
password=None, # 连接数据库
socket_timeout=None, # 执行命令的超时设置
socket_connect_timeout=None, # 连接Redis的超时设置
socket_keepalive=False, # socket特性
socket_keepalive_options=None, # 前提是socket_keepalive 为True,
socket_type=0, # 指定返回地址的协议簇,取值范围:AF_INET(IPv4)、AF_INET6(IPv6)、AF_UNSPEC(IPv4 and IPv6)
retry_on_timeout=False, # 执行失败时,如果出现TimeoutError,就会触发重新发送,只重试一次
encoding='utf-8', # 解码类型,作用于Encoder类
encoding_errors='strict', # 作用于Encoder类
decode_responses=False, # response返回的默认是bytes str,是否需要转换成执行的encoding类型
parser_class=DefaultParser, # 解析返回结果的类
socket_read_size=65536, # 作为recv的参数
health_check_interval=0, # 发送ping命令的间隔
client_name=None,
username=None # 连接redis所用的用户名
SSLConnection类在Connection的基础上额外支持的key:
ssl_keyfile=None
ssl_certfile=None
ssl_cert_reqs='required'
ssl_ca_certs=None
ssl_check_hostname=False
UnixDomainSocketConnection类支持的参数:
path=''
db=0
username=None
password=None
socket_timeout=None
encoding='utf-8'
encoding_errors='strict'
decode_responses=False
retry_on_timeout=False
parser_class=DefaultParser
socket_read_size=65536
health_check_interval=0
client_name=None
health_check_interval
PING/PONG health checks The
Redisclass and theConnectionPoolclass now support the “health_check_interval=N” option. By default N=0, which turns off health checks.Nshould be an integer, and when greater than 0, ensures that a health check is performed just before command execution anytime the underlying connection has been idle for more than N seconds. A health check is a full PING/PONG round trip to the Redis server. If a health check encounters a ConnectionError or TimeoutError, the connection is disconnected and reconnected and the health check is retried exactly once. Any error during the retry is raised to the caller. Health check retries are not governed by any other options such asretry_on_timeout. In systems where idle times are common, these health checks are the intended way to reconnect to the Redis server without harming any user data. When this option is enabled for PubSub connections, callingget_message()orlisten()will send a health check anytime a message has not been read on the PubSub connection forhealth_check_intervalseconds. Users should callget_message()orlisten()at least everyhealth_check_intervalseconds in order to keep the connection open.
成员函数解释
def from_url(cls, url, db=None, decode_components=False, **kwargs):
支持的url 类型:
- creates a normal TCP socket connection
redis://[[username]:[password]]@localhost:6379/0 - creates a SSL wrapped TCP socket connection
rediss://[[username]:[password]]@localhost:6379/0 - creates a Unix Domain Socket connection
unix://[[username]:[password]]@/path/to/socket.sock?db=0指定DB的方式有多种: 方式1: 通过querystring 中的db选项,eg: redis://localhost?db=0 方式2: 通过 redis://scheme,在path中指定db,eg:redis://localhost/0 方式3: 通过本方法中的db参数
decode_components 选项的作用是如果设置为True,对url中的username、password、path和hostname进行解码,将%xx转换为单字符,例如:'abc%20def' -> 'abc def'
if decode_components:
username = unquote(url.username) if url.username else None
password = unquote(url.password) if url.password else None
path = unquote(url.path) if url.path else None
hostname = unquote(url.hostname) if url.hostname else None
else:
username = url.username or None
password = url.password or None
path = url.path
hostname = url.hostname
kwargs 结合从url解析的参数,作为Connection的初始化参数:
# update the arguments from the URL values
kwargs.update(url_options)
....
cls(**kwargs)
def __init__(self, connection_class=Connection, max_connections=None, **connection_kwargs)
connection_class不指定的话,默认是创建正常的tcp连接
max_connection 不指定的话,默认是2 ** 31,max_connection 必须是个正数,否则会抛出ValueError异常
self._fork_lock = threading.Lock() 创建锁,因为初始化时会涉及到资源的释放self.reset()
原文是:
```
# a lock to protect the critical section in _checkpid().
# this lock is acquired when the process id changes, such as
# after a fork. during this time, multiple threads in the child
# process could attempt to acquire this lock. the first thread
# to acquire the lock will reset the data structures and lock
# object of this pool. subsequent threads acquiring this lock
# will notice the first thread already did the work and simply
# release the lock.
### `def reset(self):`
在初始化函数中调用了self.reset(),第一次初始化时调用,实际起到的是初始化以下几个成员变量的作用:
self._lock = threading.Lock()
self._created_connections = 0
self._available_connections = []
self._in_use_connections = set()
# 这个方法要在放到最后执行,表示reset逻辑已经执行完成了
# 如果不放在最后存在的问题?
# 核心逻辑在_checkpid中,先检查self.pid 和 os.getpid是否相等,如果相当,认为此进程内的连接池已经初始化完成,可以开始工作了,这个时候A线程可能会触发self._created_connections 和 self._available_connections中存在连接,但是B线程此时可能还在处理reset函数,导致A线程创建的连接被释放掉。
self.pid = os.getpid() ```
原文解释如下:
当reset()在持有_fork_lock时被调用,这个进程中的其他线程可以调用_checkpid()来比较self.pid和os.getpid()而不持有任何锁(出于性能原因)。把这个赋值作为最后一个操作,可以确保其他线程也会注意到pid的差异,并阻塞等待第一个线程释放_fork_lock。当这些线程最终获得_fork_lock时,他们会注意到另一个线程已经调用了reset(),他们会立即释放_fork_lock并继续进行.
_checkpid(self):
_checkpid()试图保证ConnectionPool的fork安全。它被所有操作线程池的ConnectionPool方法调用,如get_connection()和release()。 _checkpid()通过比较当前的进程ID和保存在ConnectionPool实例上的进程ID来确定进程是否已经fork。 当两个进程ID不一致,_checkpid() 会认为进程已经fork,并且当前运行在子进程上。子进程无法使用父进程文件描述符,(例如:sockets),因此,当子进程发现pid发生变化,需要调用reset 方法初始化当前子进程的连接池。 通过self._fork_lock来保证多线程并发的场景下,依然能够正常工作。 极端场景:
- 进程A 线程1 调用第一次_checkpid,获取到self._fork_lock
- 进程A 线程2 forks -> 进程B
- 进程B 继承了进程A的状态,也就是self._fork_lock被锁定的状态,当进程A释放self._fork_lock,进程B也不会被通知到,所以进程B的self._fork_lock一直被锁定,造成死锁。
为了消除这种可能的死锁,_checkpid() 只等待5秒去获取self._fork_lock。如果获取失败,就认为发生了死锁,rasie ChildDeadLockedError
源码:
if self.pid != os.getpid(): # python 2.7 doesn't support a timeout option to lock.acquire() # we have to mimic lock timeouts ourselves. timeout_at = time() + 5 acquired = False while time() < timeout_at: acquired = self._fork_lock.acquire(False) if acquired: break if not acquired: raise ChildDeadlockedError # reset() the instance for the new process if another thread # hasn't already done so try: if self.pid != os.getpid(): self.reset() finally: self._fork_lock.release()def get_connection(self, command_name, *keys, **options):
step1: 检查当前进程是否是fork出来的子进程,通过函数
self._checkpid来完成 step2: 从self._available_connections中获取连接,如果当前self._available_connections中没有可用的连接,创建新的连接,并将连接放入self._in_use_connections中。为了保证线程安全,用self._lock来保证self._available_connections和self._in_use_connections的线程安全。 step2.1: 检查step2中获取的线程是否有效,这里有一次失败重试的尝试。 代码如下:try: # ensure this connection is connected to Redis connection.connect() # connections that the pool provides should be ready to send # a command. if not, the connection was either returned to the # pool before all data has been read or the socket has been # closed. either way, reconnect and verify everything is good. try: # 检查当前是否有未读数据 if connection.can_read(): raise ConnectionError('Connection has data') except ConnectionError: connection.disconnect() # 重试一次 connection.connect() # 如果还有未读取的数据,抛出异常 if connection.can_read(): raise ConnectionError('Connection not ready') except BaseException: # release the connection back to the pool so that we don't # leak it # 把连接放入到池子中 self.release(connection) raise return connectiondef make_connection(self):
主要是判断当前已经创建的线程是否超过了最大线程数量 代码如下:
def make_connection(self): "Create a new connection" if self._created_connections >= self.max_connections: raise ConnectionError("Too many connections") self._created_connections += 1 return self.connection_class(**self.connection_kwargs)问题:self._created_connections 如何保证线程安全? 这个函数是在
get_connection中的with self._lock中的,所以,同样使用了self._lock保证了该变量的线程安全。
def release(self, connection):
将连接从self._in_use_connections中剔除
将连接添加到self._available_connections中
这里还有处理因为fork导致进程PID变化的问题,如果当前进程不拥有该连接(通过连接的PID完成),还需要执行以下操作:
将self._created_connections 减 1
断开该连接 connection.disconnect()
整个过程都是在self._lock 上下文中完成的。
现实环境中遇到的问题
工作中遇到问题:提示“无法验证的权限”, trace back:
File “/Users/xx/.pyenv/versions/3.6.8/lib/python3.6/site-packages/redis/client.py”, line 1606, in get return self.execute_command(‘GET’, name) File “/Users/xx/.pyenv/versions/3.6.8/lib/python3.6/site-packages/redis/client.py”, line 898, in execute_command conn = self.connection or pool.get_connection(command_name, **options) File “/Users/xx/.pyenv/versions/3.6.8/lib/python3.6/site-packages/redis/connection.py”, line 1187, in get_connection connection = self.make_connection() File “/Users/xx/.pyenv/versions/3.6.8/lib/python3.6/site-packages/redis/connection.py”, line 1225, in make_connection raise ConnectionError(“Too many connections”) redis.exceptions.ConnectionError: Too many connections
原因:Redis连接池配置的最大连接数为5个。 本地开发环境中,单个页面并发的请求数量 会大于5个,导致连接有的接口无法从redis中获取有效数据。 解决方案:调整最大连接数量为50。