Python库 调用C++库的流程

python 库创建consumer的过程:

self.consumer = PushConsumer(self.consumer_group_id)
self.consumer.subscribe(self.topic, self.callback_func)
self.consumer.start()

PushConsumer有接收三个参数,init 的原型为:

   def __init__(self, group_id, orderly=False, message_model=MessageModel.CLUSTERING):
        self._handle = dll.CreatePushConsumer(_to_bytes(group_id))
        if self._handle is None:
            raise NullPointerException('Returned null pointer when create PushConsumer')
        self._orderly = orderly
        self.set_message_model(message_model)
        self._callback_refs = []

其中:

  • group_id 表示消费者组
  • orderly 表示是否为顺序消费
  • message_model 默认为集群消费模式,还有一种消费模式是广播消费模式

dll.CreatePushConsumer的源码如下: 所属文件: src/extern/CPushConsumer.cpp

CPushConsumer* CreatePushConsumer(const char* groupId) {
  if (groupId == NULL) {
    return NULL;
  }
  DefaultMQPushConsumer* defaultMQPushConsumer = new DefaultMQPushConsumer(groupId);
  defaultMQPushConsumer->setConsumeFromWhere(CONSUME_FROM_LAST_OFFSET);

  strncpy(VERSION_FOR_PUSH_CONSUMER, defaultMQPushConsumer->version().c_str(), MAX_SDK_VERSION_LENGTH - 1);
  VERSION_FOR_PUSH_CONSUMER[MAX_SDK_VERSION_LENGTH - 1] = 0;
  return (CPushConsumer*)defaultMQPushConsumer;
}

self.consumer.subscribe的执行路径: 在python 的subscribe函数中主要调用了两个方法:

ffi_check(dll.Subscribe(self._handle, _to_bytes(topic), _to_bytes(expression)))
        self._register_callback(_on_message)

可以看到SubScribe函数实际执行了两个动作:

  • 一个是dll.Subscribe,完成消费者订阅的动作,expression表示订阅的tags
  • 另一个是注册消费回调函数

其中:dll.Subscribe方法的调用路径为:

int Subscribe(CPushConsumer* consumer, const char* topic, const char* expression) {
  if (consumer == NULL) {
    return NULL_POINTER;
  }
  ((DefaultMQPushConsumer*)consumer)->subscribe(topic, expression);
  return OK;
}

self._handler 为 defaultMQPushConsumer Subscribe 文件路径:src/extern/CPushConsumer.cpp 调用路径是:

self._handler->subscribe --> impl->subscribe   --> 
m_subTopics[topic] = subExpression; 

其中m_subTopics 的定义为: std::map<std::string, std::string> m_subTopics;

self._register_callback的执行路径: step1: 先确定是RegisterMessageCallbackOrderly 还是RegisterMessageCallback, 这取决于之前的ordely参数 step2: 将callback 添加到self.__callback_refs step3: 执行step1中注册函数,参数为self._handler,和回调函数

int RegisterMessageCallback(CPushConsumer* consumer, MessageCallBack pCallback) {
  if (consumer == NULL || pCallback == NULL) {
    return NULL_POINTER;
  }
  MessageListenerInner* listenerInner = new MessageListenerInner(consumer, pCallback);
  ((DefaultMQPushConsumer*)consumer)->registerMessageListener(listenerInner);
  g_ListenerMap[consumer] = listenerInner;
  return OK;
}

self.consumer.start()的执行路径: 执行的src/extern/CPushConsumer.cpp的 StartPushConsumer -> (DefaultMQPushConsumer*)consumer)->start() -> impl->start();

ConsumeMessageConcurrentlyService::ConsumeRequest 会实际实际执行消息的消费

try {
        status = m_pMessageListener->consumeMessage(msgs);
      } catch (...) {
        status = RECONSUME_LATER;
        LOG_ERROR("Consumer's code is buggy. Un-caught exception raised");
      }

上述方法会调用Listener的consumeMessage 方法,该方法中会触发Python中的callback函数。

进行消息的消费:

ConsumeStatus consumeMessage(const std::vector<MQMessageExt>& msgs) {
    // to do user call back
    if (m_pMsgReceiveCallback == NULL) {
      return RECONSUME_LATER;
    }
    for (size_t i = 0; i < msgs.size(); ++i) {
      MQMessageExt* msg = const_cast<MQMessageExt*>(&msgs[i]);
      CMessageExt* message = (CMessageExt*)(msg);
      if (m_pMsgReceiveCallback(m_pconsumer, message) != E_CONSUME_SUCCESS)
        return RECONSUME_LATER;
    }
    return CONSUME_SUCCESS;
  }

 private:
  MessageCallBack m_pMsgReceiveCallback;
  CPushConsumer* m_pconsumer;
};

从上述方法可以看出,如果一批消息中有一个消费失败了,就会终止后续消息的处理。