PyTorch

Pytorch

PyTorch is a machine learning library based on the Torch library, used for applications such as computer vision and natural language processing.

Resources

• https://www.learnpytorch.io/
• PyTorch cheatsheet | Official

Running PyTorch on GPU

• See Pytorch and CUDA

Tensors

Dimensions of Tensors

https://hackmd.io/@NCCU111356040/rkZiVbYcs

Linear layers

nn.Linear

Applies an affine linear transformation y = W * x + b

# Create a linear layer with 10 in features and out features
linear_layer = nn.Linear(in_features=10,
                         out_features=10)

Linear Layer Dimensions

In PyTorch, a linear layer is defined as:

nn.Linear(in_features, out_features)

• in_features is the size of each input sample.
• out_features is the size of each output sample.

Ex) For a nn.Linear layer with dimensions (4, 3):

The operation is y = W * x + b:
where ( W ) is (3, 4) and ( x ) is (4,), resulting in ( y ) of shape (3,).

Embededding Layer

• torch.nn.Embedding | A simple lookup table that stores embeddings of a fixed dictionary and size.
  • outputs a tensor with the shape (batch_size, input_size, embed_dim)

Embeddings, in the context of machine learning and natural language processing, are dense vector representations of categorical data, such as words, that capture their semantic meaning. Embeddings map words to a continuous,low-dimensional space, continuous vector of real numbers, where words with similar meanings are close to each other.

Converting a sentence into its embedding vectors with PyTorch

Guide to PyTorch’s Embedding Layer

import torch
import torch.nn as nn>
 
# Define the number of words in our vocabulary
vocab_size = 100
# Define the size of the embedding vectors
embedding_dim = 10
# Create the embedding layer
embedding = nn.Embedding(vocab_size, embedding_dim)
# Convert a single sentence into its word indices
sentence = "I love cats"
sentence_indices = [word_to_index[word] for word in sentence.split()]
# Convert the word indices into embedding vectors
embedded = embedding(torch.tensor(sentence_indices))
# Print the shape of the embedded tensor
print(embedded.shape)
# Output: torch.Size([3, 10])
# Print the values of the embedded tensor
print(embedded)
# Output:
# tensor([[ 0.0415,  0.0276, -0.0492,  0.0641,  0.0457, -0.0133, -0.0309,  0.0375,
#           0.0287, -0.0390],
#         [ 0.0201,  0.0370, -0.0458,  0.0517,  0.0208, -0.0226, -0.0390,  0.0372,
#           0.0384, -0.0276],
#         [-0.0121,  0.0381, -0.0145,  0.0666,  0.0118, -0.0485, -0.0441,  0.0414,
#          -0.0236,  0.0436]])

Embedding vs Linear Layer

What is nn.Embedding really? An embedding is basically the same thing as a linear layer (without bias term) but it does a lookup instead of a matrix-vector multiplication