AgentOS Scope OutJoin the waitlist
Scope Out wires into PropTech directories, OpenAI, and background job infrastructure — then delivers ranked intelligence to your dashboard. No third-party connectors, no configuration maze. Here is exactly what connects and why.
PIPELINE / HOW IT MOVES
The Kerfuffle crawler walks every listing page, extracts vendor entries, and persists them to discovered_products. Deduplication prevents re-ingesting known listings.
The top-N selector flags the highest-signal products for full scraping. Each gets reviews, ratings, pricing tier, and feature bullets pulled from its site and Kerfuffle listing.
The scoring engine passes scraped data to the LLM and receives replicability, market demand, revenue potential, and competitive gap scores. A weighted composite ranks every product.
Every product gets a full dossier: mission statement, suggested feature list, weakness analysis. The dashboard re-ranks instantly. Export any dossier to Markdown or PDF with one click.
48 shipped features. Kerfuffle crawling, OpenAI scoring, Inngest reliability, and Neon persistence — all wired together. Questions? Email us at sf-core-org-support-agentos-scope-out@saas-factory.ai
DATA SOURCES / INGESTION
Paste any directory URL into the Crawl dashboard. The ingestion job walks every listing page, extracts vendor names, URLs, category tags, and descriptions, then stores them for deep analysis. Kerfuffle is the starting point — the architecture supports any HTML directory you configure.
Full pagination traversal — no listings missed
Robots.txt parsed and honoured before any request
Rate-limited to 1 req/sec by default, configurable per domain
After the skim, Scope Out visits each product's own website and its Kerfuffle listing. It pulls star ratings, review text, pricing tier signals, and feature copy — storing everything in product_scrape_data linked to the discovered product. Site-specific CSS/XPath extractors handle known domains; a generic heuristic covers anything new.
AI LAYER / OPENAI
Every scraped product passes through an LLM pipeline that evaluates four dimensions and composes a ranked dossier. Token usage and cost are tracked per crawl run.
LLM evaluates data model complexity, real-time requirements, third-party integration count, and AI needs against the AgentOS stack. Returns a 1–10 score and written rationale.
Combines pricing tier with category market size (from the PropTech lookup table) and review-implied adoption. Outputs an estimated ARR range for a competitor entering that space.
BACKGROUND INFRASTRUCTURE / INNGEST
Every crawl, scrape, scoring run, and scheduled re-crawl is an Inngest background function. That means retries, observability, and fan-out are handled without you building queue infrastructure.
Directory crawl job Fires on paste-URL trigger; handles pagination and deduplication
Deep-scrape batch One Inngest event per product; runs concurrently, capped for rate safety
AI scoring pipeline Chained function: scrape data in — four dimension scores + dossier out
DATA LAYER / NEON + DRIZZLE
Name, URL, source directory, category tags, description, and status. The root record every downstream table links to.
Raw ratings, review count, individual review text, pricing tier, feature bullets, and scraped_at timestamp. Retained for 90 days by default.
COMPARISON VIEW / DASHBOARD
Select 2–4 products from the ranked dashboard and open a comparison table. All four dimension scores, pricing tier, category, review count, and weakness themes align in columns so you can distinguish between products with similar composite scores.
All scored dimensions in aligned columns
Pricing tier and review count side-by-side
Weakness themes surfaced from negative review mining
OUTPUT / WHAT YOU GET
Export any individual product dossier as a formatted Markdown file or PDF. Export the full ranked table as a CSV for offline review. Each of the three dossier artefacts — mission statement, feature list, weakness analysis — has its own copy-to-clipboard button in Markdown.
Single-dossier export: Markdown or PDF
Full ranked table: CSV from the dashboard toolbar
Per-artefact copy: mission, features, weakness in Markdown
ACCESS / SECURITY
Access is restricted to an email allowlist enforced at sign-in. Any attempt from an unlisted address is rejected immediately with an 'access restricted' message. There is no public sign-up flow — the root URL redirects authenticated users to the dashboard and unauthenticated users to sign-in.
Hardcoded at the middleware level. Unlisted emails never reach the app.
Root URL bypasses any marketing page and routes straight to sign-in. No crawlable auth pages for external actors.
FAQ / INTEGRATIONS
Exponential back-off on 429 / 503 responses
Duplicate-run guard prevents concurrent crawls of the same URL
Pricing normaliser Maps raw price signals to free / freemium / low / mid / high / enterprise tiers
Plugin registry Site-specific extractors for known high-value domains, fallback heuristics for everything else
Error & retry UI Failed scrapes (403, timeout, robots block) show on the dashboard with one-click retry
Mines negative reviews exclusively. The LLM identifies the top 3 recurring complaints and maps them to opportunity themes — 'no mobile app', 'pricing too high', 'poor API'.
Combines review volume (log scale), recency, average star rating, and positive/negative sentiment ratio into a single 1–10 demand signal with a written explanation.
AI cost per crawl run is tracked and displayed on the run detail page. A configurable threshold triggers an in-app alert if a single run exceeds your budget.
Weekly re-crawl cron Monitored directories re-crawled automatically; new products flagged with badges
Weekly digest email Cron emails top discoveries and score movements to the allowlisted team
Four dimension scores plus weighted composite. Versioned — every re-score snapshots the previous result so you can track market movement over time.
Mission statement, suggested feature list, and weakness analysis stored as JSONB. Each artefact is copy-to-clipboard ready in Markdown from the dossier page.
Every crawl run is logged with status, duration, and product counts. AI token usage and cost per run are stored separately — monthly spend is always visible.
Paste directly into the SaaS platform builder
In-app notifications fire when a crawl completes or a shortlisted product's score shifts significantly. A weekly email digest delivers the top new discoveries and score movements to all allowlisted team members — configurable per user.
In-app: crawl complete with product count summary
In-app: shortlisted product score drop alert
Email: weekly top-5 new discoveries and top-3 by score
In-app: AI cost threshold exceeded warning
Robots.txt is fetched and parsed before any domain is touched. Blocked domains are logged to the crawl run record.
