--- title: Pozify emoji: "🏋️" colorFrom: green colorTo: blue sdk: gradio sdk_version: "6.17.3" python_version: "3.10" app_file: app.py fullWidth: true short_description: Small-model workout form review from short videos. tags: - gradio - computer-vision - pose-estimation - fitness - video-analysis - llama-cpp - track:backyard - sponsor:openbmb - sponsor:openai - sponsor:nvidia - sponsor:modal - achievement:offgrid - achievement:welltuned - achievement:offbrand - achievement:llama - achievement:sharing - achievement:fieldnotes --- # Pozify Pozify is a small-model workout form coach for people who want to train at home but still need clear, trustworthy feedback. It is built for users who avoid gyms because they are far away, too crowded, intimidating, or too expensive to replace with a 1:1 personal trainer. Upload a short workout video and Pozify turns it into a structured, grounded form-review report: - exercise detected - reps counted - variation and issue markers - annotated video and clips - grounded coach summary - verifier-backed confidence and safety notes The app is built as a small-model pipeline, not a giant end-to-end model. It combines transparent pose analysis, a trained exercise router, exercise-specific rule logic, deterministic knowledge-card retrieval, and a small summary model. Pozify is not a medical device. It does not diagnose injuries, claim injury prevention, or replace a qualified trainer, clinician, or physical therapist. ![Pozify product demo](docs/assets/demo.png) ## Hackathon Snapshot - Track: `Backyard AI` - Core user impact: affordable at-home workout feedback without needing a gym or private coach - Submission format: `Gradio Space` - Build Small fit: every runtime model used by Pozify is under the `32B` cap - Demo video: `ADD_PUBLIC_DEMO_LINK` - Social post: `ADD_PUBLIC_SOCIAL_POST_LINK` - Hugging Face Space: [build-small-hackathon/Pozify](https://huggingface.co/spaces/build-small-hackathon/Pozify) - Team repo: [tihado/Pozify](https://github.com/tihado/Pozify) - Default coach-summary model: `build-small-hackathon/pozify-coach-summary1` ## The Problem Most beginner and intermediate gym users do not need a full-time trainer. They need a fast second set of eyes: - "Am I doing the right exercise?" - "How many clean reps did I actually complete?" - "Is this a valid variation or a real issue?" - "What should I fix next session?" Today, that feedback is often inaccessible: - gyms can be far away or inconvenient - crowded spaces make people self-conscious about training in public - many users are afraid of doing an exercise wrong and being judged - private coaching is effective, but too expensive for regular use Pozify makes that feedback accessible from a short video, with a pipeline users can inspect instead of a black-box answer they just have to trust. ## Why It Stands Out Pozify is not a generic chatbot and not a vague video captioner. It is a grounded movement-analysis pipeline: - computer vision extracts pose landmarks - a tiny trained router identifies the exercise - deterministic logic counts reps and tracks issues - knowledge cards keep the coaching language exercise-specific - a small language model turns structured JSON into a coach summary - a verifier catches unsafe or ungrounded summary output That design is the core product difference: structured evidence first, language second. ## Sponsor Stack Primary sponsor tools used in this build: - `Hugging Face Spaces` for the app surface - `Hugging Face Inference` for cloud small-model runtime - `Modal` for coach-summary training and publishing workflows - `OpenAI Codex` for implementation support and hackathon build velocity Sponsor-fit highlights: - `Modal`: used to prepare data, train, evaluate, merge, and publish the coach-summary model - `OpenAI Codex`: used as the coding copilot during implementation and iteration - `Hugging Face`: used across the product surface, cloud inference path, model hosting, and Space deployment ## Why Pozify Fits Build Small Pozify matches `Backyard AI` because it solves a real everyday problem with a small, practical, personal tool. It also strongly matches the broader Build Small philosophy: - local-first and modular architecture - transparent model boundaries instead of one giant opaque system - per-component models all under the `32B` limit - useful enough for repeated day-to-day use, not just a tech demo This gives Pozify a practical consumer use case that still feels very "Build Small": local-first, inspectable, modular, and cheap enough to run on real-world hardware budgets. ## What Pozify Delivers For each uploaded workout clip, Pozify produces: - detected exercise and confidence - rep-by-rep analysis JSON - valid variation markers versus real issues - annotated output video - grounded coach summary with fixes and next-session plan - provider, model, and summary source metadata in the UI The UI also makes it obvious whether the coach summary came from: - `hf_inference` - a local merged model - or a conservative fallback ## Product Flow ```text video + user profile -> video QC -> MediaPipe pose extraction -> pose cleaning -> exercise router -> exercise-specific rep counter -> per-rep analysis -> variation detection -> issue markers -> annotated video renderer -> grounded coach summary -> verifier -> final report ``` Supported router labels: - `squat` - `push_up` - `shoulder_press` - `unknown` ## Model Stack | Component | Model or method | Trained here? | Runtime | | ----------------------- | -------------------------------------------------------- | ----------------------- | ---------------------------------------------- | | Pose extraction | MediaPipe Pose Landmarker Lite | No | CPU / MediaPipe delegate | | Exercise router | PyTorch BiLSTM over 30-frame pose windows | Yes | Torch | | Router baseline | scikit-learn `HistGradientBoostingClassifier` | Yes | CPU fallback/reference | | Rep counting | Exercise-specific state machines | No ML | CPU | | Issue markers | Transparent rules over per-rep metrics | No ML | CPU | | Coach summary | `build-small-hackathon/pozify-coach-summary1` by default | Fine-tuned merged model | HF Inference, local Transformers, or llama.cpp | | Coach-summary fine-tune | LoRA / merged checkpoint pipeline on Modal | Yes | Local merged-model path recommended | | Verifier | Deterministic safety and grounding checks | No ML | CPU | The trained router is intentionally tiny: | Artifact | Count | | ------------------------------- | --------: | | BiLSTM router trainable params | 182,796 | | Router input features per frame | 237 | | Window length | 30 frames | | Output classes | 4 | ## Run The App Locally This repo uses a `src/` layout, but `uv` is configured with `package = false`. Run it with: ```bash uv sync uv run python app.py ``` Then open `http://127.0.0.1:7860`. ### Mock vs Real Mode By default: - if no video is provided, Pozify uses mock mode - if a real video is uploaded, Pozify runs the full analysis pipeline Force mock mode: ```bash POZIFY_MOCK_MODE=1 uv run python app.py ``` Force real mode: ```bash POZIFY_MOCK_MODE=0 uv run python app.py ``` If you already have the MediaPipe task file locally: ```bash POZIFY_MEDIAPIPE_POSE_MODEL=/path/to/pose_landmarker_lite.task \ POZIFY_MOCK_MODE=0 \ uv run python app.py ``` ## Coach Summary Runtime Options ### 1. Fine-tuned coach model The app defaults to the fine-tuned coach-summary model: ```bash export POZIFY_COACH_SUMMARY_MODEL=build-small-hackathon/pozify-coach-summary1 uv run python app.py ``` Pozify tries `chat_completion` first and falls back to `text_generation` when Hugging Face reports that the repo is not a chat model. The deterministic fallback summary remains enabled if hosted inference is unavailable or the model output fails validation. Recommended if you want the live Space or local demo to behave predictably during judging. ### 2. Use the fine-tuned merged model locally Download the merged repo locally, then point Pozify at it: ```bash export POZIFY_COACH_SUMMARY_LOCAL_MODEL_DIR=/absolute/path/to/merged_model export POZIFY_COACH_SUMMARY_BASE_MODEL=Qwen/Qwen3-14B export POZIFY_COACH_SUMMARY_ADAPTER_ID=build-small-hackathon/pozify-coach-summary1 uv run python app.py ``` This is the simplest way to use `build-small-hackathon/pozify-coach-summary1` today without adding a dedicated inference endpoint. ### 3. Base cloud model override If you need the previous base-model runtime: ```bash export POZIFY_COACH_SUMMARY_MODEL=Qwen/Qwen3-14B uv run python app.py ``` ### 4. llama.cpp Pozify can send the coach-summary prompt to a local `llama-server` that exposes the OpenAI-compatible `/v1/chat/completions` endpoint. Example: ```bash llama-server \ --model /path/to/qwen3-14b-instruct.gguf \ --ctx-size 4096 \ --n-gpu-layers 99 \ --host 127.0.0.1 \ --port 8080 ``` Then: ```bash POZIFY_COACH_SUMMARY_PROVIDER=llama_cpp \ POZIFY_COACH_SUMMARY_MODEL=local-qwen3-14b-gguf \ POZIFY_LLAMA_CPP_BASE_URL=http://127.0.0.1:8080 \ POZIFY_COACH_SUMMARY_MAX_TOKENS=700 \ uv run python app.py ``` ### Useful environment variables | Variable | Purpose | | -------------------------------------- | -------------------------------------------------------- | | `POZIFY_ROUTER_DEVICE` | Override router device, for example `cpu` or `cuda`. | | `POZIFY_SPACES_GPU_DURATION` | `spaces.GPU` duration in seconds, default `120`. | | `POZIFY_COACH_SUMMARY_PROVIDER` | `hf_inference`, `local_transformers`, or `llama_cpp`. | | `POZIFY_COACH_SUMMARY_MODEL` | Coach model id or llama.cpp model alias. | | `POZIFY_COACH_SUMMARY_LOCAL_MODEL_DIR` | Prefer a local merged/model directory for coach summary. | | `POZIFY_COACH_SUMMARY_BYPASS_VERIFIER` | Keep model output even when verifier fails. | ## Exercise Router Training Run the full router training and publish flow: ```bash uv run modal run scripts/exercise_router_modal.py \ --stage all \ --repo-id build-small-hackathon/pozify-exercise-router ``` Step-by-step: ```bash uv run modal run scripts/exercise_router_modal.py --stage ingest uv run modal run scripts/exercise_router_modal.py --stage features uv run modal run scripts/exercise_router_modal.py --stage train-baseline uv run modal run scripts/exercise_router_modal.py --stage train-temporal uv run modal run scripts/exercise_router_modal.py --stage evaluate uv run modal run scripts/exercise_router_modal.py --stage publish --repo-id build-small-hackathon/pozify-exercise-router ``` The active router artifact is `temporal.pt`; the baseline is retained for comparison and fallback. ## Coach Summary Training Build the grounded SFT dataset: ```bash uv run python scripts/build_coach_summary_sft_dataset.py ``` Run the full coach-summary Modal flow: ```bash uv run modal run scripts/coach_summary_modal.py \ --stage all \ --epochs 2 \ --style-weight 0.2 \ --repo-id build-small-hackathon/pozify-coach-summary1 ``` Step-by-step: ```bash uv run modal run scripts/coach_summary_modal.py --stage prepare-data uv run modal run scripts/coach_summary_modal.py --stage train --epochs 2 --style-weight 0.2 uv run modal run scripts/coach_summary_modal.py --stage evaluate --limit 5 uv run modal run scripts/coach_summary_modal.py --stage merge uv run modal run scripts/coach_summary_modal.py --stage publish-merged --repo-id build-small-hackathon/pozify-coach-summary1 ``` Important runtime note: - the default coach model is `build-small-hackathon/pozify-coach-summary1` - Hugging Face hosted inference may still reject a repo or produce invalid JSON, so the conservative fallback summary stays enabled - for the most predictable fine-tuned inference path, use `POZIFY_COACH_SUMMARY_LOCAL_MODEL_DIR` ## Generated Artifacts Each run creates `runs//` with: - `manifest.json` - `user_profile.json` - `video_manifest.json` - `pose_sequence.json` - `exercise_classification.json` - `reps.json` - `rep_debug.json` - `rep_analysis.json` - `variation.json` - `issue_markers.json` - `annotated_video.mp4` - `coach_summary.json` - `verification.json` - `final_report.json` JSON artifacts are validated before they are written. The final report records: - analysis mode - pose source - knowledge-card provenance - coach summary provider/model/source - verifier status and bypass flags ## Docs For the deeper technical write-up, training notes, and data workflow, see [docs/01-docs-index.md](docs/01-docs-index.md). ## Development Checks ```bash uv run ruff check uv run python -m compileall src scripts tests app.py uv run python -m unittest discover -s tests ``` Run the real MediaPipe fixture smoke test only when the fixture is available: ```bash POZIFY_RUN_REAL_POSE_TESTS=1 \ uv run python -m unittest tests.test_pose_steps.PoseStepTests.test_real_sample_mov_extracts_pose_landmarks ``` ### Contributors - 🚀 [@nvti](https://github.com/nvti) - 🌿 [@honghanhh](https://github.com/honghanhh) - 🔧 [@NLag](https://github.com/NLag) - ✨ [pnhneee](https://github.com/ctpnheee)