Language model engineering
AetherMind / Forge-1
A compact decoder-only language-model stack for constrained consumer GPUs.
Role: Architecture and training implementation
Reading the project context and repository evidence.
01
challenge
Training experiments on consumer hardware need explicit memory budgets. A model architecture that looks compact on paper can still exceed the available memory once activations, optimizer state and batches are included.
02
approach
Design a roughly 125-million-parameter variant targeting 4 GB VRAM and pair it with memory-aware training. Keep architecture, training, inference and evaluation in one inspectable implementation.
03
solution
The stack implements RMSNorm, Grouped Query Attention, RoPE, SwiGLU and weight tying. Training supports FP16 mixed precision, gradient checkpointing and gradient accumulation to work within constrained hardware.
04
impact
The CV documents the implemented architecture and the 4 GB VRAM design target. It does not provide a final benchmark or measured training throughput, so those outcomes are not claimed here.