PyTorch gives importance to ease of experimentation over static graph optimization. It is built for the developer productivity, flexibility, and debuggability.

• Eager execution
• Define-by-run dynamic computation graph
• Python-first API with a high-performance C++ backend
• Clear Python stack traces for easier debugging

• Static graph optimization is not as important to PyTorch as developer productivity. This gap is partially filled by PyTorch 2.x (torch.compile).

Because of its adaptability, efficiency, and robust ecosystem, PyTorch is a deep learning framework that is extensively used in both research and production.

• Dynamic computation graph
• NumPy-like, Pythonic syntax
• Powerful GPU acceleration via CUDA
• Extensive use in research and production systems

• Research and experimentation
• Quick prototyping
• Custom or novel architectures
• Learning deep learning concepts
• NLP and advanced model development

• Large-scale production systems
• Embedded or mobile deployment
• TPU-intensive workloads
• Enterprise ML pipelines

• torch → tensors and mathematical operations
• torch.nn → neural network layers and modules
• torch.optim → optimization algorithms
• torch.autograd → automatic differentiation
• torch.utils.data → datasets and dataloaders

• Load data
• Define model
• Define loss function
• Define optimizer
• Training loop
• Evaluation

• Storage: 1D contiguous memory buffer
• Sizes (shape)
• Strides: Steps (in elements) to move across dimensions
• Offset: Starting position in storage
• dtype
• device

x = torch.randn(3, 4)
x.storage()
x.stride()

• torch.tensor([1, 2, 3])
• torch.zeros(3, 4)
• torch.ones(2, 2)
• torch.empty(5, 5)
• torch.eye(3)
• torch.arange(0, 10, 2)
• torch.linspace(0, 1, 5)

• torch.rand(3, 3)
• torch.randn(3, 3) # Normal distribution

• import numpy as np
• a = np.array([1, 2, 3])
• t = torch.from_numpy(a)

• t.shape
• t.dtype
• t.device
• t.requires_grad

• torch.float32
• torch.float64
• torch.int64
• torch.bool

• t.float()
• t.long()
• t.to(torch.float32)

• a + b
• a - b
• a * b
• a / b
• torch.add(a, b)

• torch.matmul(a, b)
• a @ b
• a.T
• torch.mm(a, b)

• torch.sum(a)
• torch.mean(a)
• torch.max(a)
• torch.min(a)

• Dimensions align from right
• Size must match or be 1

• t[0]
• t[:, 1]
• t[1:3]
• t[t > 0]

• indices = torch.tensor([0, 2])
• t[indices]

• t.view(2, 3)
• t.reshape(2, 3)
• t.squeeze()
• t.unsqueeze(1)

• x = torch.randn(3, 4) # shape (3,4), strides (4,1)
• Move across columns: 1 step
• Move across rows: 4 steps
• Transpose: y = x.t() # shape (4,3), strides (1,4)
• No data duplication

• view
• reshape (occasionally)
• transpose
• squeeze/release

• clone

• Contiguous tensor: memory arranged in row-major order
• Necessary for many CUDA kernels
• Check with x.is_contiguous()
• Call .contiguous() after transpose/permute before heavy operations

• device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
• t = t.to(device)

• Dynamic automatic differentiation engine
• Records tensor operations during forward pass
• Constructs a Directed Acyclic Graph (DAG)
• reverse-mode autodiff executed via .backward()
• Non-leaf tensors have .grad_fn, leaf tensors accumulate .grad

• Created directly by user
• requires_grad=True
• .grad populated after backward
• Example: x = torch.randn(3, requires_grad=True)

• Result of an operation
• Has grad_fn
• .grad NOT retained by default
• Example: y = x * 2
• To retain gradient: y.retain_grad()

• Each operation creates a Function node
• Example: x → MulBackward → MeanBackward → z
• Graph is freed after .backward() by default
• Use retain_graph=True to keep the graph

• z.backward() # equivalent to z.backward(torch.ones_like(z))

• y.backward(gradient=torch.ones_like(y))

• Gradients accumulate by default
• Example: loss.backward() twice → grads doubled
• Must clear gradients: optimizer.zero_grad()

• Every tensor has a version counter
• In-place modification increases version
• Autograd checks version consistency
• Example Error: x.add_(1) breaks graph for y.backward()

• Returns a tensor sharing storage
• Stops gradient tracking
• Example: y = x.detach()

• Context manager to temporarily disable graph construction
• Example: with torch.no_grad(): y = model(x)

• Stronger than no_grad
• Disables version counters
• Faster inference

• x.requires_grad = True → y = x*2 inherits requires_grad
• Exceptions: integer tensors cannot require gradients
• Operations with constants inherit from tensors

• Autograd saves intermediate tensors during forward pass
• Needed for backward computation
• Memory-heavy ops: attention, large matmuls, conv layers

• Custom CUDA ops
• Memory optimization
• Numerical stability

class MyFunc(torch.autograd.Function):
    @staticmethod
    def forward(ctx, x):
        ctx.save_for_backward(x)
        return x ** 2

    @staticmethod
    def backward(ctx, grad_output):
        x, = ctx.saved_tensors
        return grad_output * 2 * x

y = MyFunc.apply(x)

• Enable graph during backward: loss.backward(create_graph=True)
• Used in meta-learning, gradient penalties, second-order optimizers
• ⚠️ Memory explosion risk

• x.register_hook(lambda grad: grad * 0.5)

• Forward hooks
• Backward hooks

• Using .data: x.data *= 2 # breaks autograd silently
• In-place ReLU: nn.ReLU(inplace=True)
• Forgetting model.train() / model.eval()
• Mixing numpy and torch tensors

• Use logsumexp
• Avoid softmax + log
• Use fused losses like CrossEntropyLoss

• Graph creation during forward pass
• Gradient flow during backward pass
• Memory saved per operation
• Where synchronization happens

• DAG of tensor operations
• Built dynamically
• Freed after .backward() unless retain_graph=True
• Disable gradients with: with torch.no_grad(): y = model(x)

• Parameter registration
• Submodule tracking
• Mode switching (train / eval)
• State serialization
• Hook infrastructure
• Device & dtype propagation

• Attributes assigned as nn.Parameter are registered
• Stored in self._parameters
• Accessible via model.parameters(), state_dict(), and optimizer

class Model(nn.Module):
    def __init__(self):
        super().__init__()
        self.w = nn.Parameter(torch.randn(10))

• Plain tensors: self.w = torch.randn(10)
• Python lists: self.layers = [nn.Linear(10, 10)]
• Use nn.ModuleList, nn.ModuleDict, nn.ParameterList for collections

• Tensors not optimized but saved in state_dict
• Move automatically with .to(device)
• Used in BatchNorm, running statistics, masks

self.register_buffer('running_mean', torch.zeros(10))

• output = model(x) calls __call__
• __call__ does: pre-forward hooks, calls forward, post-forward hooks, handles autograd
• Never call forward() directly

• Dropout
• BatchNorm
• LayerNorm (partially)

• Disable gradients
• Change requires_grad

• Parameters
• Buffers
• Names reflect module hierarchy

model.state_dict().keys()

• Linear → Kaiming uniform
• Conv → Kaiming normal

• Feature extraction
• Debugging gradients
• Visualization

• self.linear = nn.Linear(10, 10); self.linear2 = self.linear
• Same weights
• One set of gradients

• RNNs
• Siamese networks
• Tied embeddings

• self.backbone
• self.head

• nn.ModuleDict

• class Block(nn.Module): ...

• Print named_parameters()
• Check gradient norms
• Verify device placement
• Assert shapes

for n, p in model.named_parameters():
    print(n, p.grad.norm())

• Storing tensors from forward pass
• Keeping graph references
• Hooks not removed

handle.remove()

• Parameter vs buffer
• Why ModuleList matters
• How __call__ works
• Why eval() matters
• Why pickling models is bad

• Linear: nn.Linear(in_features, out_features)
• Convolution: nn.Conv2d(in_channels, out_channels, kernel_size)
• Pooling: nn.MaxPool2d(2), nn.AvgPool2d(2)
• Normalization: nn.BatchNorm1d(num_features), nn.LayerNorm(shape)

• nn.ReLU()
• nn.LeakyReLU(0.01)
• nn.Sigmoid()
• nn.Tanh()
• nn.Softmax(dim=1)

• Regression: nn.MSELoss(), nn.L1Loss()
• Classification: nn.CrossEntropyLoss() # includes Softmax, nn.BCELoss(), nn.BCEWithLogitsLoss()

• lr
• weight_decay
• momentum

import torch.optim as optim
optim.SGD(model.parameters(), lr=0.01)
optim.Adam(model.parameters(), lr=0.001)
optim.AdamW(model.parameters())
optim.RMSprop(model.parameters())

for epoch in range(epochs):
    model.train()
    for x, y in dataloader:
        optimizer.zero_grad()
        pred = model(x)
        loss = criterion(pred, y)
        loss.backward()
        optimizer.step()

# Evaluation
model.eval()
with torch.no_grad():
    ...

from torch.utils.data import Dataset, DataLoader

class MyDataset(Dataset):
    def __init__(self, X, y):
        self.X = X
        self.y = y

    def __len__(self):
        return len(self.X)

    def __getitem__(self, idx):
        return self.X[idx], self.y[idx]

dataloader = DataLoader(dataset, batch_size=32, shuffle=True)

• Key Parameters: batch_size, shuffle, num_workers, pin_memory

torch.save(model.state_dict(), 'model.pth')
model.load_state_dict(torch.load('model.pth'))
model.eval()

# Full model save (Not Recommended)
torch.save(model, 'model.pth')

• nn.init.xavier_uniform_(layer.weight)
• nn.init.kaiming_normal_(layer.weight)

• Dropout: nn.Dropout(p=0.5)
• Weight Decay: optim.Adam(model.parameters(), weight_decay=1e-4)
• Early Stopping: Monitor validation loss, stop when loss increases

• ReduceLROnPlateau
• CosineAnnealingLR

from torch.optim.lr_scheduler import StepLR
scheduler = StepLR(optimizer, step_size=10, gamma=0.1)
scheduler.step()

• Freeze Layers: for param in model.parameters(): param.requires_grad = False
• Replace Head: model.fc = nn.Linear(512, num_classes)

• Conv Output Size: O = (W − K + 2P) / S + 1
• Key Concepts: Receptive field, Stride, Padding, Channels vs spatial dims

• nn.RNN(input_size, hidden_size)
• nn.LSTM(input_size, hidden_size)
• nn.GRU(input_size, hidden_size)
• Shapes: (batch, seq_len, features)

• Core Blocks: Embedding, Multi-head self-attention, Feedforward, Residual + LayerNorm
• nn.TransformerEncoderLayer(d_model, nhead)

from torch.cuda.amp import autocast, GradScaler
scaler = GradScaler()

with autocast():
    loss = criterion(pred, y)
scaler.scale(loss).backward()
scaler.step(optimizer)
scaler.update()

• Common Errors: Shape mismatch, Forgetting .zero_grad(), Using .item() inside graph, Wrong loss for task
• Debug Tools: print(t.shape), torch.isnan(t).any()

• Use torch.no_grad() for inference
• Use .contiguous()
• Avoid Python loops
• Batch operations