We are releasing the official technical report for DeepX Embedding v1.0, a state-of-the-art Vietnamese embedding model engineered specifically for legal document retrieval and high-precision semantic search.
Developed by the DX Tech team, DeepX v1.0 combines a novel linear attention mechanism (Gated DeltaNet-2) with Hyperloop weight sharing, achieving superior retrieval accuracy while maintaining O(n) time and memory efficiency for long sequences.
Executive Summary
On the industry-standard Zalo Legal Text Retrieval benchmark, DeepX Embedding v1.0 achieves an nDCG@10 of 0.8162 (and Recall@10 of 0.9638). This establishes a new State-of-the-Art (SOTA) among sub-1B parameter models (< 1B params), outperforming the previous benchmark record (0.7813) by +3.49 points.
Key Technical Innovations & DX Contributions:
-
- O(n) Linear Attention (GDN-2): Replaces standard softmax attention with gated delta-rule recurrence. This maintains constant VRAM usage per token regardless of sequence length, enabling fast inference across long documents (trained up to 8,192 tokens).
-
- Per-Loop LoRA (DX Contribution): Solves the depth collapse inherent in recursive weight sharing. Each loop iteration receives its own low-rank adapter (rank 16) applied across entire 5-layer blocks simultaneously (Q,K,V,O, gate, up, down), giving every compute pass a unique representation identity.
-
- Rotary Depth Embedding / RoDE (DX Contribution): Introduces a novel positional signal that encodes loop iteration depth directly into attention. By applying a second rotation on Q and K based on iteration index (using 8 rotary pairs), RoDE informs the attention mechanism “how deep” in the network it currently is.
-
- Two-Stage Retrieval & Matryoshka Embeddings: Produces dual outputs—a 1536-d single vector for fast ANN search (with 256-d Matryoshka truncation preserving ~96% quality) and ColBERT token vectors (T×128-d) for precise late-interaction reranking.
Read the full technical report embedded below for complete architectural breakdowns, dataset mixtures, and benchmarking methodology.