Composio · AI Product Ops take-home

Which of these 100 apps could become an agent toolkit today, and what's blocking the rest?

A research agent (Gemini on Vertex AI plus Google Search and a crawler) checked each app's auth, credential gating, API surface and MCP availability, citing the source for every claim. A second model then fact-checked every record, a crawler checked every cited URL, and a different model family (Claude) re-researched a random 20-app sample blind to measure accuracy. Patterns first, table below.

The patterns

What the 100 rows say when clustered. Every number is computed from apps.json.

Buildability verdict by category

Sorted by share ready today. Hover a segment for the apps in it.

How a developer gets credentials, by category

Darker = more open. Grey = no public API.

Primary auth (what an integration would use)

Main blocker (apps not ready today)

MCP server availability

Where to act

The same 100 apps, split into what to do next.

Easy wins: build now

    Buildable with friction

      Needs outreach / not feasible

        All 100 apps

        Click a row for the notes, evidence links with quotes, and what the verifier changed. ✎ = a field changed by the verifier or corrected by the cross-check.

        #AppCategoryPrimary authAll auth AccessAPIMCPVerdictBlockerSources
        Definitions used by the agent (the same rubric was given to researcher and verifier)
        Access
        self-serve free: anyone can sign up and get working API credentials for free (free tier, dev account, sandbox or open-source). self-serve trial: only during a time-limited trial. paid plan: API needs a paid subscription. approval required: free to start, but production needs app review, a developer-token approval or admin approval. partner / sales: partner program, enterprise contract or contact-sales. no public API.
        Verdict
        ready now: public docs plus self-serve creds, buildable this week. friction: buildable with a hurdle (paid plan, trial, app review, narrow API, or an open-source CLI that must be sandboxed). needs outreach: blocked on a partnership or sales deal. not feasible: nothing callable.
        MCP
        official: published or hosted by the vendor. community only: third-party servers only.

        The agent: what it does, and where a human was needed

        A Python pipeline (run.py) runs each stage over all 100 apps in parallel and caches per-app JSON, so any stage can be re-run on its own. Models: Gemini 3 Flash does the research; Gemini 2.5 Pro, a different model, acts as the adversarial verifier. Both run on Vertex AI with Google Search grounding.

        agent · Flash + Search

        1. Discover

        Searches for the vendor's auth, pricing, API-reference and MCP pages.

        code

        2. Validate URLs

        An HTTP check on every candidate. Invented paths (404) are dropped before the agent reads anything.

        code + agent

        3. Read & research

        The crawler fetches the live pages; the agent writes notes where every claim has a URL and a verbatim quote.

        agent · schema

        4. Extract

        Notes become a strict JSON schema with fixed enums, so the rows can be clustered.

        code, then Pro + Search

        5. Verify

        The crawler re-fetches every cited URL (alive? official? is the quote really on the page?) and lints for contradictions. Then Pro re-derives every field, using those flags.

        independent model

        6. Spot-check

        A random sample of 20 apps (2 per category) is re-researched blind by a different model family (Claude) from the live docs. Every disagreement with Gemini is re-checked against the vendor page and decided with a cited quote.

        By the numbers

        Where a human was needed

          Verification: how we know it's right

          Three independent loops, and each catches a different kind of error. Hits and misses are shown as they are.

          Loop 3: blind independent check on a 20-app sample

          Loop 1: citation crawler (all 100 apps)

          first passafter verifier
          The crawler fetches every cited URL itself. "Quote found" = at least half of the quote's 4-word phrases appear on the page (readable pages only; JavaScript-rendered or bot-walled pages can't be checked this way).

          Loop 2: what the verifier changed

          Hits and misses (20-app sample)

          Known limits

            Run it

            Source repo: github.com/cipher-v/composio_assignment. Needs Python 3.12, a GCP project with Vertex AI enabled, and gcloud auth application-default login.

            uv venv && uv pip install -r requirements.txt
export COMPOSIO_RESEARCH_GCP_PROJECT=your-project
python run.py research --ids 4,31   # pass 1 on two apps (Attio, Google Ads)
python run.py all                   # full pipeline for all 100: research, crawl-check, verify, re-check, merge, analyze, page
python run.py review                # build the human spot-check sheet, then export human_labels.json
python run.py merge && python run.py score && python run.py analyze && python run.py page

            For agents: data/apps.json (per-app records with evidence), data/patterns.json, data/accuracy.json, llms.txt.