# Agneya Tharun > Portfolio of Agneya Tharun, a Carnegie Mellon University student working in cybersecurity, machine learning, and robotics. This file is the plain-text version of https://agneya.me: the site itself is an animated single page that builds itself in front of the visitor, so everything it conveys is written out here. - GitHub: https://github.com/Vort3xed - LinkedIn: https://www.linkedin.com/in/agneya-tharun - Email: agneyat2 [at] gmail.com ## Featured projects ### d4rt clone A single feed-forward model that turns ordinary video into 4D geometry: per-pixel depth, long-range 3D point tracking, and camera pose, all recovered from one unified query interface. Tags: PyTorch, ViT, 3D Vision, CUDA. How it works, as shown on the page: 1. Input video. A short clip is played, then T frames are sampled and resized to a square for the model. 2. Encoder. A Vision Transformer cuts each frame into patch tokens, then interleaves per-frame and global self-attention to mix space and time. 3. Global scene representation. The encoder distills the whole clip into F, a latent scene representation computed once and reused for every query. 4. Pointwise decoder. Each query (u, v, t_src, t_tgt, t_cam) cross-attends into F to predict a 3D point. A custom 5-layer pose head with 9D-rotation SVD recovers the camera. 5. 4D reconstruction. Querying every pixel, every frame, yields a live colored point cloud that the visitor can orbit. Results: - 64x less compute (single L40S vs a 64-TPU cluster). Reproduced the paper's benchmarks on one NVIDIA L40S GPU. Matched the paper's ViT-B Sintel depth loss and stayed competitive with the 1B-parameter VGGT model on Sintel PC L1 geometric-mean loss (1.635 vs 1.582) at 7x fewer parameters. - -53% trajectory error (custom pose-decoder head). Engineered a Pi3/LoGeR-style 5-layer pose head with 9D-rotation SVD orthogonalization to fix the pose-estimation bottleneck, cutting ATE by 53% and beating the published ViT-B baseline on camera pose by 26%. - 2.4x train, 2,400 fps (throughput and memory). Vectorized query generation (256x speedup), pre-resized fp16 dataset caches (35x less disk), AMP mixed precision, and CUDA-allocator tuning enabled full ViT-B training in 46 GB on one consumer GPU. 50 clips/sec at fp16, 3.5 GB peak for inference. ### Incremental 3D Reconstruction (TreeHacks 2026) A live phone feed becomes a continuously growing 3D point cloud in the browser, with text-promptable 3D object detection. Built on the pi^3 / Pi3X feed-forward geometry model at TreeHacks 2026. Tags: Pi3X, SAM3, three.js, FastAPI. How it works, as shown on the page: 1. Capture. A phone streams its walkthrough over a secure WebSocket. The server decodes frames into a bounded queue and a GPU worker pulls chunks from it. 2. Chunking. A background thread groups frames into chunks of 16 with a 6-frame overlap, so consecutive chunks share enough view to be stitched together. 3. Pi3X plus prior. Each chunk runs through Pi3X. The previous chunk's poses, depth, and rays are injected into the first six slots as a prior for a consistent reconstruction. 4. Sim(3) stitch. A masked Umeyama solve recovers one rotation, scale, and translation that drops each new chunk precisely into the global cloud. 5. SAM3 to 3D boxes. Type a prompt and SAM3 segments it in 2D. The mask is lifted into 3D through Pi3X's point map and wrapped in an oriented box, then placed in the scene. 6. Live reconstruction. The real reconstructed office: 150,000 colored points with deduplicated 3D detections across five prompts (chair, monitor, desk, table, trash can). Results: - Real time, in browser (phone to server to viewer). A three-process system: a phone sender streams frames over secure WebSockets to a FastAPI server with a bounded, drop-on-overflow queue, and a three.js viewer renders the cloud as it grows. - Sim(3) Umeyama stitching (chunked and prior-conditioned). Frames run through Pi3X in 16-frame chunks with 6-frame overlap; the previous chunk's poses, depth, and rays are injected as a prior, and each new chunk is aligned into the global frame by a masked Umeyama SVD. - Text-promptable 3D boxes (SAM3 to 3D OBBs). SAM3 segments prompted objects on keyframes; the 2D masks are lifted into 3D through Pi3X's dense point maps and fitted to oriented bounding boxes, deduplicated across frames and streamed to the viewer. ### attackbot Two Unitree G1 humanoids fight in simulation. The ego learns by reinforcement learning to punch, kick, stay upright, and get back up. It trains in 4,096 arenas on one DGX Spark. Every frame on the page is real simulator footage, and the last step replays a recorded fight in 3D from the simulator's own body poses on the G1's geometry. Tags: mjlab, MuJoCo-Warp, PPO, PyTorch. How it works, as shown on the page: 1. Arenas. mjlab on MuJoCo-Warp runs 4,096 arenas on one DGX Spark: about 106,000 environment steps per second, 50 Hz control. 2. Motion tracker (layer 1). A PPO tracking policy follows 37 retargeted martial-arts clips on the G1 under physics: state plus reference frames in, 29 joint targets at 50 Hz out. It reaches 4.8 degrees mean joint error on the roundhouse and falls 2-9% of the time while shoved; NVIDIA's pretrained general tracker fell on 4 of 5 fight clips. 3. 32-D latent (layer 2). The tracker is distilled (DAgger) into an encoder, a decoder, and a prior. The latent lives on a 32-D unit sphere; the decoder is then frozen, and every skill above acts by picking a point on the sphere. The student tracks 4.2-4.4 degrees per joint against the teacher's 4.8. 4. Expert drills (layer 3). One frozen expert per move (punch, side kick, roundhouse, get-up, approach), each trained by PPO in a drill where that move is the only way to score: a fight against a scripted opponent for the punch, an isolated kick drill for the side kick, the reference replayed as a 65% prior and annealed to zero for the roundhouse, 1,726 real fall poses for the get-up, and reaching striking range from 2-6 m under fire for the approach. 5. Compose. The move is commanded, not chosen (a shuffled round-robin plus GETUP and APPROACH latches). A small PPO-trained head rotates the commanded expert's latent inside a cone of 31 degrees (punch, side kick, approach), 50 degrees (roundhouse), or 0 (get-up). Unbounded, the residual had grown to 12x the expert and made the experts decorative; the cone keeps them load-bearing. The head is trained against a league of seven frozen checkpoints of earlier policies, one per arena, with a self-play mode. 6. The whole stack. Three policies on one data path: s (351-D: proprioception plus opponent) into the experts and the head, z = z_c + delta on the 32-D sphere into the frozen decoder D(s_p, z) with s_p the 93-D proprioception, 29 joint targets out to a PD controller at 50 Hz. Layer 1's tracker is distilled into layer 2's decoder; only the layer-3 head is still learning. 7. 3D replay. The same 16-second self-play round replayed from the simulator's body poses on the real G1 meshes, with strike markers at the limb that landed. White is the ego, red is the opponent. Results (bands over six scorecard reads): - 106k steps/s (4,096 arenas on one GB10). mjlab on MuJoCo-Warp runs 4,096 arenas on one DGX Spark, about 106,000 environment steps per second. All 265 training runs fit on one desk-side machine. - Recovered after a fall: 0% to 70-74%. The get-up expert was trained on clips that were mostly face-up, while real fight falls are 32% face-down. Routing it through the decoder it was trained against took recovery from 0% to 28% with no retraining, and spawning it from 1,726 real fall poses lifted fight recovery from 31% to 59%. - Roundhouse lands per attempt: 37% to 63-68%, with 920-1,152 N peak impact. A thrown roundhouse lands 84-86% of the time. Punches went from 45% to 69-77%, side kicks from 45% to 53-59%, three-move chains from 12% to 26-33%, time spent down from 38% to 8.7-9.9%, and 9 of 15 scorecard criteria pass (from 7), each judged against bootstrap confidence intervals and replicate spread. - Three strikes plus get-up versus RoboStriker's punches only. RoboStriker is the published G1 boxing recipe this project builds on; it fights with punches. This policy also lands side kicks and roundhouse kicks, gets up after a fall, and chains moves, all trained on one desk-side box. (RoboStriker was deployed to real hardware; attackbot is simulation only.) ## Selected work - Kerberos (1st, NVIDIA Hackathon; 1st, Ascend Track). A search-and-rescue command center that runs entirely on one DGX Spark. Phones and robots stream camera frames in. The box tracks each pose with DPVO, fuses every agent onto one metric map, flags casualties with a VLM, and streams AR guidance and a live dashboard back. DPVO, Isaac Sim, DGX Spark. Demo: https://youtu.be/x9bd5u4GTMo - Colossus (2nd, HackCMU 2026). Asymmetric multiplayer: one player embodies a skyscraper-sized robot on a Quest 2 while up to eight jetpack raiders on laptops try to bring it down. A Node server runs Rapier physics for an endless Manhattan whose buildings shatter bay by bay. Three.js, Rapier, WebXR. Demo: https://www.youtube.com/watch?v=8IrUBAuz7Pw - llama.cpp Vulkan Q2_0 Kernel. A merged contribution to llama.cpp's Vulkan backend that implements the Q2_0 ternary format, so 1.58-bit BitNet-style models run fully on any Vulkan GPU instead of falling back to CPU. A block of 128 weights stores one fp16 scale plus 32 bytes of 2-bit codes (about 2.1 bits per weight). Five GLSL shaders wired in at about 32 registration points, verified bit-for-bit against the ggml reference by a 959-line CPU simulator (262,677 checks, 0 failed); Ternary-Bonsai-8B offloads fully to Vulkan on an Intel Arc GPU at 31.5 tokens/sec generation and 52 tokens/sec prompt. Merged as PR #32, +1,128 lines. C++, GLSL, Vulkan, ggml. PR: https://github.com/PrismML-Eng/llama.cpp/pull/32 - GPU Compute Core over UART. A fabricated 2x2 matrix-multiplier chip, taped out through TinyTapeout. It reads two matrices over UART, computes the product in a state machine, and returns the result on serial and parallel pins. Verilog, UART, Sky130. Repo: https://github.com/Vort3xed/tt10-verilog-template-agneya - CMU AI Poker Bot (1st, Jump Trading). A bot that approximates Nash equilibrium for a Texas Hold'em variant. It originally used precomputed hand-rank tables and Monte Carlo methods, then a model trained to detect the top 10 best bots and deploy specific countermeasures against them. Python, MCCFR, RL. Private. - Vantage (3rd, VentureHacks). A geospatial disaster-response tool that pulls satellite imagery, detects each roof with a vision-language model, scores damage 0 to 100, and plans contractor routes with PDF reports. FastAPI, Moondream, React. Demo: https://drive.google.com/file/d/1se_nSOGAaOfEEBSxi-zl2dXnEoA2vA5n/view - AVScan2Vec2. A dual-transformer autoencoder that embeds a file's antivirus-scan results into a vector, enabling malware classification, clustering, and nearest-neighbor lookup over a Qdrant index. PyTorch, Transformers, Qdrant. Repo: https://github.com/Vort3xed/AVScan2Vec - Cognit (1st, NexHacks). Uses a Tobii eye-tracker to verify a user actually read a document, then mints a verifiable on-chain certificate of completion from a generated comprehension quiz. FastAPI, Tobii, Solidity. Private. - Sparrows. A platform for schools to manage schedules, clubs, and events, with a student side offering academic resources and an in-browser code compiler that runs on the edge via WebAssembly. Next.js, WASM, Cloudflare. Live: https://schoolnest.org - Perspectron. Turns any screen into a touchscreen. A phone camera tracks hand movements and sends deltas over WebRTC, with precomputed homographies from ArUco markers to warp the cursor position and compensate for perspective distortion. MediaPipe, WebRTC, Electron. Demo: https://youtu.be/0ePwr6E6DE4 - Thryving. A mental-health webapp for teens: keep a journal, build widget flowcharts to map your journey, and get AI sentiment analysis on entries, with auth and subscriptions. Next.js, Gemini, Stripe. Live: https://thryving.app - DuckHunt Remastered (1st, HackCMU 2025). Turns your phone into a light gun to play Duck Hunt on a shared screen with friends, streaming phone motion with near-zero latency over WebRTC. WebRTC, Socket.io, Node. Live: https://gymote-server.fly.dev - TWDNE. Every subdirectory you visit is generated on the fly by an LLM: visit /anything and get a plausible page for it, rendered live. Next.js, Groq, Supabase. Live: https://thispagedoesnotexist.com ## Contact Email agneyat2 [at] gmail.com, or reach out on GitHub (https://github.com/Vort3xed) or LinkedIn (https://www.linkedin.com/in/agneya-tharun). The page signs off with "let's build stuff :)".