WeightManager

class WeightManager(device: Union[DevManager, int] = 0)

Represents weight manager reference for Module. This is a wrapper of WeightManager used for passing objects without any substantive methods.

For large language models, like Qwen, if each weight within the models is allocated a block of device memory to store the weight values, it may lead to substantial memory consumption. To save device memory, modules can share their weight memory with a weight manager when performing multi-tasks within a module, or across multi-modules with the same weight values on the same Houmo device.

Notes

When creating a weight manager object, the actual weight manager is not instantiated immediately. This object is initialized only when tcim_lite.runtime.Module.load() or tcim_lite.runtime.load() is called with the provided Option object, which contains the WeightManager. This deferred instantiation allows for more efficient resource allocation during the model loading process.

Examples

The following example shows how to create a weight manager and load models with the same weight manager:

import tcim_lite as tcim

# Create an option with weight manager
weight_manager = tcim.runtime.WeightManager(0)
# Create a Option object to set configurations for loading models
option = tcim.runtime.Option(weight_manager)
# Load models with the same option
module_1 = tcim.runtime.load("model_name.hmm", option=option)
module_2 = tcim.runtime.load("model_name.hmm", option=option)

Constructor for the WeightManager class.

Parameters:

device (int or DevManager) -- int: The logical ID of the Houmo device on which the weight is stored. DevManager: The selected device logic id and backends