o kg^@sbddlZddlZddlmZz ddlmZWdSey0edGdddejj ZYdSw)N) FusedRMSNormzdimepscs&t||_tt||_dS)a Initialize the RMSNorm normalization layer. Args: dim (int): The dimension of the input tensor. eps (float, optional): A small value added to the denominator for numerical stability. Default is 1e-6. Attributes: eps (float): A small value added to the denominator for numerical stability. weight (nn.Parameter): Learnable scaling parameter. N)super__init__rnn Parametertorchonesweight)selfrr __class__C/mnt/petrelfs/gaopeng/qinqi/lumina2_open-sorce/models/components.pyr s zRMSNorm.__init__cCs$|t|djddd|jS)z Apply the RMSNorm normalization to the input tensor. Args: x (torch.Tensor): The input tensor. Returns: torch.Tensor: The normalized tensor. T)keepdim)r rsqrtpowmeanr)rxrrr_norms$ z RMSNorm._normcCs|||}||jS)z Forward pass through the RMSNorm layer. Args: x (torch.Tensor): The input tensor. Returns: torch.Tensor: The output tensor after applying RMSNorm. )rfloattype_asr )rroutputrrrforward*s zRMSNorm.forward)r) __name__ __module__ __qualname__intrrrr __classcell__rrrrr s r) warningsr torch.nnr Zapex.normalizationrr ImportErrorwarnModulerrrrs