AIFit

Overview

AIFit helps product teams evaluate AI feature ideas across four dimensions:

AI Fit
Commercial Upside
Risk Burden
Evidence Readiness

The tool generates structured recommendations, human review workflows, and validation plans to support responsible AI product decisions.

The Problem

Many AI product discussions focus on capability and feasibility:

Can the model do it?
Is the technology available?
How quickly can we ship it?

Less attention is often given to:

whether AI is the right solution;
how the feature may influence human decisions;
what risks should be reviewed before launch;
how the feature should be validated.

I built AIFit to explore a more systematic approach to evaluating new AI product features.

The Solution - AIFit

AIFit combines structured feature inputs, risk-specific guidance, and LLM evaluation to generate a decision-support tool.

The goal is not to automate product decisions, but to help teams ask better questions before committing resources.

System Architecture

System flow from feature idea to risk-aware decision report.

Key Design Decisions

(i) Risk-specific Evaluation

AIFit identifies a dominant risk type and adapts review and validation guidance accordingly.

Examples of risk types include:

False Validation
Emotional Vulnerability
Financial Manipulation
Fairness / Bias
Privacy / Data Sensitivity

(ii) Structured outputs instead of free-form advice

AIFit breaks the evaluation into specific fields such as risk type, risk themes, review package, validation method, metrics, move-forward criteria, and stop/redesign signals. This makes the output easier to inspect, compare, and challenge.

(iii) Human-in-the-loop By Default

The tool generates recommendations, but all outputs are framed as decision support rather than automated decisions. Product, domain, legal, policy, or compliance experts remain responsible for reviewing outputs before launch decisions.

(iv) Explainable Outputs

AIFit provides:

Risk classification – Identifies the dominant product risk that should guide evaluation and governance.
Risk themes – Highlights the specific failure modes or concerns associated with the feature.
Risk rationale – Explains why the feature was assigned a particular risk classification.
Review workflows – Recommends who should review the feature, what artifacts to inspect, and when review should occur.
Validation workflows – Suggests how the feature should be tested, what scenarios should be covered, and what evidence is needed before launch.

Together, these outputs make AIFit recommendations easier to interpret, review, and challenge.

Example Outputs

Evaluation Summary Output

Evaluation summary showing recommendation, scoring, and risk classification.

Build Boundaries Output

Defines useful scope and risky build patterns.

Human Review Workflow Output

Human review workflow generated for the selected risk type.

Validation Workflow Output

Validation workflow generated for the selected risk type.

Lessons Learned

Building AIFit reinforced several key ideas:

Prompt design is product architecture.
Risk evaluation requires domain-specific guidance.
Human review and validation are product features, not compliance afterthoughts.
Explainability is often more valuable than additional model complexity.

Future Work

Planned improvements include:

Separating risk classification from evaluation generation – Improve explainability and make risk assignment easier to debug and validate independently.
Externalizing the risk guidance library – Move risk-specific guidance out of the prompt and into configurable files to simplify maintenance and future expansion.
Support for additional risk categories – Expand the framework as new recurring risk patterns emerge during testing and real-world usage.
Prompt regression testing – Evaluate prompt changes against a library of representative use cases to detect improvements or unintended degradations in output quality.
Improved explainability for scoring – Provide clearer reasoning for how AI Fit, Commercial Upside, Risk Burden, and Evidence Readiness scores are determined.
Model comparison and evaluation – Compare outputs across different LLMs to assess consistency, reasoning quality, and recommendation robustness.

Tech Stack

Python
Streamlit
OpenRouter
GitHub
Streamlit Community Cloud