AI ENGINEERING EXPERIMENT
Testing AI Virtual Try-On for Fashion Commerce
What a broad model benchmark taught us about quality, speed, cost, and production readiness
EXECUTIVE SUMMARY
A Commerce Problem, Not Just an Image Problem
Shoppers need a trustworthy way to preview garments on themselves before purchase. Our research therefore looked for more than a visually pleasing demo. We needed commerce-quality output that represents the product credibly, returns quickly enough for a storefront interaction, and costs little enough to operate at scale.
| CURRENT BENCHMARK WINNER Gemini 3.1 produced the strongest overall visual quality and category consistency in our current tests. Flux VTO was the best speed-and-cost alternative. |
What We Tested
We evaluated general image models, dedicated hosted VTO APIs, and open-source models running on self-hosted GPUs. The experiments covered tops, jackets, bottoms, dresses, ethnic garments, simple poses, difficult poses, flat-lay garments, and garments photographed on people.
Contents
| Section | Topic |
|---|---|
| 1 | Problem, success criteria, and experiment design |
| 2 | How we used AI and Codex to run the research |
| 3 | Model landscape and what each class can do |
| 4 | Visual evidence across hosted and self-hosted models |
| 5 | Complete benchmark sheet and winner analysis |
| 6 | Lessons, production recommendation, and next steps |
EXPERIMENT DESIGN
How We Made the Comparison Useful
The first large benchmark produced many images but also revealed a testing trap: identical payloads do not always create a fair comparison. Some models accept natural-language instructions; others depend on category labels, segmentation, masks, pose data, or provider-specific quality controls. We improved the method in later rounds by respecting those differences while keeping the person and garment inputs fixed.
Success Criteria
| Criterion | Question |
|---|---|
| Visual appeal | Would a shopper accept this as a credible fashion image? |
| Garment fidelity | Are silhouette, colour, print, trim, embroidery, and structure preserved? |
| Identity preservation | Does the person still look like the input person? |
| Fit realism | Do sleeves, waistlines, hems, hands, and drape look physically plausible? |
| Consistency | Does quality hold across tops, bottoms, dresses, jackets, and ethnic garments? |
| Operations | Are latency, cost, failure rate, and hosting effort acceptable? |
How We Used AI to Run the Experiment
Codex acted as the engineering and research partner throughout the work. It searched model documentation and repositories, decided whether a model was worth testing, prepared model-specific payloads, called hosted APIs concurrently, configured GPU environments, assembled self-hosted pipelines, tracked failures and timings, and generated side-by-side comparison sheets. Human review remained essential: we selected the business goal, supplied access and datasets, and judged whether an output was trustworthy enough for commerce.
- Research: Read official docs and repositories; mapped inputs, licenses, hosting requirements, and tuning options.
- Execution: Ran repeatable tests, retries, parameter sweeps, seed comparisons, and concurrent API calls.
- Evaluation: Created visual comparison boards and converted observations into benchmark tables and recommendations.
MODEL LANDSCAPE
Three Different Ways to Solve Virtual Try-On
General Image Models
Gemini 3.1 and Flux 2 are not dedicated VTO engines. They behave more like image editors with broad visual understanding. This gives them strong prompt following and the ability to rebuild lighting, styling, and composition. It also creates a product-fidelity risk because the model may improve, simplify, or invent a detail.
Hosted Specialist VTO Models
Flux VTO, Vertex VTO-001, FASHN TryOn Max, FASHN v1.6, Kolors, CatVTON, and Leffa expose person-plus-garment workflows. They reduce infrastructure work and usually return results in seconds. Their controls vary: some accept prompts, while others rely mainly on category, garment photo type, segmentation mode, seed, resolution, or provider defaults.
Self-Hosted Open-Source Models
FASHN v1.5, IDM-VTON, StableVITON, and FastFit gave us full runtime visibility and avoided per-image API dependence. In return, they introduced GPU selection, checkpoint downloads, Python compatibility, preprocessing, memory limits, and inference optimisation. A model can be correctly installed yet still produce weak output if its expected masks, human parsing, DensePose, crop, or category inputs are wrong.
| IMPORTANT DISTINCTION A polished image is not always the most faithful try-on, and a dedicated VTO model is not automatically production-ready. The right choice depends on whether the product values presentation, strict garment fidelity, speed, or controllability most. |
What Changed During the Research
- From common prompts to model-aware payloads: We stopped assuming every endpoint understood the same instructions.
- From one output to repeated evidence: We compared categories, seeds, crops, modes, resolutions, and self-hosted environments.
- From model hype to commerce evidence: We judged outputs against product detail and shopper trust, not repository screenshots.
VISUAL EVIDENCE
One Pair Revealed Very Different Model Behaviour
The denim-skirt case is useful because the garment has a clear silhouette, button, fly, pockets, seam placement, fading, and distressed marks. The same person-and-garment pair exposed how differently models balance exact preservation and attractive reconstruction.
- Gemini 3.1: Produced the most finished fashion image, but its reconstruction behaviour can change small product details.
- Flux VTO and Vertex: Kept the task grounded as a try-on, but visual richness and garment detail were less consistent.
- FASHN, Kolors, and IDM-VTON: Showed credible structure in some cases, yet quality varied with category and input preparation.
- Flux 2: Created attractive edits, but as a general image model it did not provide the same strict VTO guarantees.
HOSTED MODEL FINDINGS
Specialist Models Were Strong, but Not Consistently Stronger
What the Hosted API Round Taught Us
- FASHN v1.6: Covered all main categories, but mixed lower-body fidelity and an estimated $0.075/image did not beat Gemini.
- Kolors: Returned several fast, credible results, but sometimes flattened garment detail.
- CatVTON and Leffa: Produced usable examples, yet latency and inconsistency weakened their hosted results.
- FASHN TryOn Max: Quality/2K preserved detail; fast/1K reduced time. Neither was a consistent Gemini replacement.
SELF-HOSTED FINDINGS
Owning the Model Also Means Owning the Pipeline
Self-hosting was valuable because it separated model capability from provider packaging. It also showed why repository demos can be misleading: the visible model is only one part of the system. The preprocessing stack, checkpoints, GPU architecture, precision, input crop, and inference settings can determine the final result.
Model-by-Model Lessons
- FASHN v1.5: Ran successfully on Kaggle, but one test took about 6.5 minutes on the selected T4x2 environment. The gap from advertised hosted speed points to runtime optimisation and hardware utilisation, not only model quality.
- IDM-VTON: Ran on a self-hosted GPU with denoise steps, seeds, and crop sweeps. It handled some garments well, but the skirt case remained inconsistent and sometimes changed the garment class.
- StableVITON: The first smoke tests did not fairly exercise the model because the complete human parsing, DensePose, agnostic image, agnostic mask, and garment mask pipeline was not yet present.
- FastFit: Ran on stronger GPU hardware with recommended settings, yet outputs drifted heavily from the requested garment. More compute did not repair model-task mismatch.
| ENGINEERING LESSON A successful inference call proves that software runs. It does not prove that the full VTO system is correctly prepared or commercially useful. |
COMPLETE BENCHMARK SHEET
Major Models Tested and Their Current Position
The table combines direct API tests, focused head-to-head runs, and self-hosted experiments. Exact numbers are shown only where a clean record was captured; other rows use qualitative operational evidence.
| Model | Access | Controls / coverage | Observed result | Decision |
|---|---|---|---|---|
| Gemini 3.1 | Hosted general image | All tested categories; prompt-led | Best visual appeal and consistency; occasional invented detail | OVERALL QUALITY WINNER |
| Flux VTO | Hosted dedicated VTO | All main categories; seed/prompt | Faster and cheaper; less polished on fine detail | SPEED/COST WINNER |
| Flux 2 Pro/Max | Hosted general edit | Prompt + person + garment | Attractive results; higher cost; not strict VTO | Secondary baseline |
| Vertex VTO-001 | Hosted dedicated VTO | Person + product image | Stable VTO workflow; weaker presentation in tested pairs | Monitor |
| FASHN TryOn Max | Hosted dedicated VTO | Mode, resolution, prompt controls | Strong specialist; quality mode slower; not consistently better | Strong contender |
| FASHN v1.6 | Fal-hosted VTO | Top, bottom, one-piece | Broad completion; mixed category fidelity; ~$0.075/image | Useful specialist |
| Kolors | Fal-hosted VTO | Category-based VTO | Fast examples; inconsistent detail preservation | Conditional |
| CatVTON | Fal-hosted / open | Category and basic quality settings | Lightweight idea; slow/uneven hosted results in our tests | Research only |
| Leffa | Fal-hosted / open | Category and quality controls | Some credible fits; identity and silhouette drift | Research only |
| IDM-VTON | Self-hosted open source | Prompt, category, masks, steps, seed | Good baseline potential; preprocessing-heavy; lower-body issues | Needs pipeline work |
| FASHN v1.5 | Self-hosted open source | Person, garment, category | Functional; ~6.5 min in Kaggle T4x2 test | Optimise before retest |
| StableVITON | Self-hosted open source | Full parsing/pose/mask stack | Initial output invalid as fair benchmark without preprocessing | Incomplete test |
| FastFit | Self-hosted research | Person, garment, category/description | Ran on GPU; output was not commercially relevant | Do not prioritise |
| HOW TO READ THIS The top two are highlighted because they won different dimensions: Gemini 3.1 for presentation-ready output; Flux VTO for lower latency and cost. This is the current result, not a permanent universal ranking. |
CONTROLLED HEAD-TO-HEAD
Gemini 3.1 Versus Flux VTO Across 11 Pairs
This was our cleanest repeated benchmark. Both models received the same person-and-garment pairs, while prompts were adapted to each model's behaviour. Flux VTO used seed 42 in later runs because it gave the best tested balance of fit and speed.
| Criterion | Gemini 3.1 | Flux VTO | Winner |
|---|---|---|---|
| Visual appeal | 9.0 | 7.6 | Gemini |
| Garment detail accuracy | 8.4 | 7.9 | Gemini, slight |
| Identity preservation | 8.5 | 8.4 | Similar |
| Realism of fit | 8.6 | 7.9 | Gemini |
| Consistency across categories | 8.5 | 7.8 | Gemini |
| Speed | 7.5 | 9.0 | Flux VTO |
| Cost | 7.8 | 8.8 | Flux VTO |
Measured Averages
| Metric | Gemini 3.1 | Flux VTO | Difference |
|---|---|---|---|
| Average generation time | 12.36s | 9.39s | Flux 2.97s faster |
| Average cost per image | ~$0.0500 | ~$0.0416 | Flux ~$0.0084 cheaper |
Why Gemini Won This Benchmark
Gemini won because shoppers judge the complete image. It reconstructed lighting, composition, face quality, and garment presentation together. Specialist models were often more literal, but literal output still failed the commerce bar when details looked flat, fit boundaries were awkward, or the image felt unfinished.
DECISION AND NEXT STEPS
What We Would Build Today
| RECOMMENDATION Use Gemini 3.1 as the current premium-quality baseline and Flux VTO as the lower-latency, lower-cost route. Keep specialist and self-hosted research focused on measurable gaps rather than testing models only because they are labelled VTO. |
A Practical Production Strategy
- Quality route: Send complex dresses, rich prints, embroidery, draped garments, and premium experiences to Gemini 3.1.
- Fast route: Use Flux VTO when response time and cost matter more, especially for simpler garments and high-volume interactions.
- Validation: Add automated checks for one visible person, usable framing, product coverage, and obvious output corruption before returning an image.
- Evidence loop: Store inputs, outputs, latency, cost, settings, and reviewer scores so upgrades can be compared against the same baseline.
What We Learned About Applying AI to Commerce
The most important result was a repeatable way to turn a fast-moving AI landscape into an engineering decision. AI helped research the market, run experiments, configure infrastructure, analyse failures, and package evidence. Human commerce judgement decided whether outputs were trustworthy for shoppers and defensible for merchants.
What Comes Next
- Expand the benchmark: Add body shapes, skin tones, poses, photography styles, and difficult occlusions.
- Score fidelity explicitly: Use garment-detail checklists for buttons, zips, logos, prints, borders, cuts, and lengths instead of relying only on visual appeal.
- Complete specialist pipelines: Retest open models only after their required preprocessing is production-grade.
- Test business impact: Measure engagement, add-to-cart behaviour, retries, and trust feedback.
Conclusion
In our current experiments, Gemini 3.1 is the overall winner for commerce-ready output, while Flux VTO is the best speed-and-cost alternative. This does not invalidate dedicated VTO. It shows that fidelity, identity, realism, polish, latency, cost, and shopper trust must be judged together.
Evidence base: internal Shootless benchmark runs completed June-July 2026. Scores are manual visual assessments of the tested inputs and configurations.