ez_transfer.layers

activations

easytransfer.layers.activations.gelu(x)

Gaussian Error Linear Unit. Original Implementation of the gelu activation function in Google Bert repo when initially created.

For information: OpenAI GPT's gelu is slightly different (and gives slightly different results): 0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))) Also see https://arxiv.org/abs/1606.08415

easytransfer.layers.activations.gelu_new(x)

Gaussian Error Linear Unit. This is a smoother version of the RELU. Original paper: https://arxiv.org/abs/1606.08415 :param x: float Tensor to perform activation.

Returns:x with the GELU activation applied.

easytransfer.layers.activations.swish(x)

easytransfer.layers.activations.relu(x)