Open Scouts

Create AI scouts that continuously search the web and notify you when they find what you're looking for.

About

Open Scouts is an AI-powered monitoring platform that lets you create "scouts" - automated tasks that run on a schedule to continuously search for and track information. Whether you're looking for new restaurants near you, monitoring AI news, or tracking any other updates, scouts work 24/7 to find what you need and notify you when they discover it.

Tech Stack

• Next.js 16 (with App Router & Turbopack)
• React 19
• TypeScript
• Tailwind CSS v4
• Supabase (Database + Auth + Edge Functions)
• pgvector (Vector embeddings for semantic search)
• Firecrawl SDK (@mendable/firecrawl-js)
• OpenAI API (AI Agent + Embeddings)
• Resend (Email Notifications)

Getting Started

Prerequisites

• Node.js 18+
• bun (default), npm, or pnpm
• Supabase account (supabase.com)
• OpenAI API key (platform.openai.com)
• Firecrawl API key (firecrawl.dev)
• Resend API key (resend.com) - for email notifications
• Google Cloud Console account (for Google OAuth - optional)

1. Clone and Install

git clone https://github.com/leonardogrig/open-scout
cd open-scout
bun install  # or: npm install / pnpm install

2. Create Supabase Project

1. Go to supabase.com
2. Create a new project
3. Wait for the project to finish provisioning

3. Enable Required Extensions

In your Supabase Dashboard:

1. Go to Database → Extensions
2. Search for and enable:
   • pg_cron (for scheduled jobs)
   • pg_net (for HTTP requests from database)
   • vector (for AI-powered semantic search on execution summaries)
   • supabase_vault (for secure credential storage - usually enabled by default)

4. Set Up Environment Variables

Create a .env file in the root directory by copying the example file:

cp .env.example .env

Then fill in your actual values in the .env file.

The .env.example file contains all required environment variables with detailed instructions and direct links for where to obtain each API key.

5. Run Database Setup

First, link your Supabase project (required for syncing secrets):

bunx supabase login        # Login to Supabase CLI (one-time)
bunx supabase link --project-ref <your-project-ref>  # Find ref in Supabase Dashboard URL

Then run the setup script:

bun run setup:db  # or: npm run setup:db / pnpm run setup:db

This will:

• Create all required tables (scouts, scout_executions, scout_execution_steps, etc.)
• Add user authentication support (user_id columns, Row Level Security)
• Enable real-time subscriptions
• Set up vector embeddings for AI-generated execution summaries
• Configure the scalable dispatcher architecture (pg_cron + pg_net + vault)
• Automatically store your Supabase URL and service role key in the vault
• Set up cron jobs for scout dispatching and cleanup
• Sync Edge Function secrets from your .env file (OPENAI_API_KEY, FIRECRAWL_API_KEY, RESEND_API_KEY)

Note: The setup script will check if the required extensions (vector, pg_cron, pg_net) are enabled. If not, follow the on-screen instructions to enable them in the Supabase Dashboard, then run the script again.

6. Set Up Authentication

Open Scouts uses Supabase Auth for user authentication, supporting both email/password and Google OAuth.

Enable Email/Password Auth (Enabled by Default)

1. Go to Supabase Dashboard → Authentication → Providers → Email
2. Ensure "Enable Email Provider" is toggled on
3. Configure email templates as needed in Authentication → Email Templates

Enable Google OAuth (Optional but Recommended)

1. Create Google OAuth Credentials:

   • Go to Google Cloud Console
   • Create a new project or select existing one
   • Navigate to APIs & Services → Credentials
   • Click Create Credentials → OAuth client ID
   • Choose "Web application" as Application type
   • Add authorized JavaScript origins:
     • http://localhost:3000 (development)
     • https://your-domain.com (production)
   • Add authorized redirect URIs:
     • https://<your-project-ref>.supabase.co/auth/v1/callback
   • Copy the Client ID and Client Secret

2. Configure in Supabase:

   • Go to Supabase Dashboard → Authentication → Providers → Google
   • Toggle "Enable Google Provider"
   • Paste your Client ID and Client Secret
   • Save

7. Deploy Edge Functions

Deploy the scout execution agent and email functions to Supabase Cloud:

bunx supabase functions deploy scout-cron
bunx supabase functions deploy send-test-email

Note: Secrets (OPENAI_API_KEY, FIRECRAWL_API_KEY, RESEND_API_KEY) are automatically synced when you run setup:db. If you need to update them manually:

bunx supabase secrets set OPENAI_API_KEY=sk-proj-...

8. Set Up Resend (Email Notifications)

Email notifications are sent to your account email when scouts find results.

1. Create a Resend account at resend.com
2. Get your API key from the Resend dashboard
3. Add to .env and run setup:db again to sync, or set manually:

   bunx supabase secrets set RESEND_API_KEY=re_...

4. Verify a custom domain at resend.com/domains to send to any email

Important - Free Tier Limitations:

• Without a verified domain, Resend only sends to your Resend account email
• Free tier includes 3,000 emails/month (100/day limit)

Testing Email Setup:

1. Go to Settings in the app
2. Click Send Test Email to verify the configuration
3. Check your inbox for the test email

9. Firecrawl Configuration

Open Scouts uses Firecrawl for web scraping and search. There are two ways to configure it:

Option A: Standard API Key (Recommended for Contributors)

This is the simplest setup - all users share a single API key:

1. Sign up at firecrawl.dev
2. Get your API key from the dashboard
3. Add to your .env file:

   FIRECRAWL_API_KEY=fc-your-key-here

4. Set the edge function secret:

   npx supabase secrets set FIRECRAWL_API_KEY=fc-your-key-here

Option B: Partner Integration (For Production Deployments)

If you're deploying Open Scouts for multiple users and want per-user API key management:

1. Contact Firecrawl to obtain a partner key
2. Set your partner key in .env:

   FIRECRAWL_API_KEY=your-partner-key

3. Set the edge function secret:

   npx supabase secrets set FIRECRAWL_API_KEY=your-partner-key

How Partner Integration Works:

• When users sign up, a unique Firecrawl API key is automatically created for them
• Each user's usage is tracked separately
• Keys are stored securely in the user_preferences table
• If a user's key fails, the system automatically falls back to the shared partner key
• Users can view their connection status in Settings → Firecrawl Integration

Benefits:

• Better usage tracking per user
• Ability to revoke individual user keys
• Automatic key provisioning on signup
• Self-healing: invalid keys are detected and fallback kicks in

Note: The partner integration is fully backwards compatible. If you don't have a partner key, the system works exactly like Option A with a shared key.

10. Run the Development Server

bun run dev  # or: npm run dev / pnpm run dev

Open http://localhost:3000 to see the app.

How It Works

User Authentication Flow

1. Public Home Page: Users can browse the landing page without signing in
2. Create Scout: When a user types a query and hits Enter, they're prompted to sign in
3. Sign In/Sign Up: Users can authenticate via email/password or Google OAuth
4. Continue Flow: After authentication, the scout creation continues automatically
5. User Isolation: Each user only sees and manages their own scouts

Scout System

1. Create a Scout: Define what you want to monitor (e.g., "Scout for any recent Indian restaurants near me" or "Scout for any AI news")
2. AI Agent Setup: The system automatically configures search queries and strategies
3. Set Frequency: Choose how often to run (hourly, every 3 days, weekly)
4. Configure Notifications: Add your email in Settings to receive alerts when scouts find results
5. Continuous Monitoring: The dispatcher checks every minute and triggers due scouts individually
6. AI Summaries: Each successful execution generates a concise one-sentence summary with semantic embeddings
7. Get Notified: Receive email alerts when scouts find new results (if email is configured)
8. View Results: See all findings with AI-generated summaries in real-time on the scout page

Manual Execution

Click the "Run Now" button on any scout page to trigger execution immediately without waiting for the cron.

Email Notifications

When scouts find results, you'll automatically receive email alerts at your account email:

• Automatic: Emails are sent only when scouts successfully find results
• Rich Content: Beautiful HTML emails with scout results and links
• Test: Use the "Send Test Email" button in Settings to verify setup

Email Service: Powered by Resend (free tier includes 3,000 emails/month)

Note: On Resend's free tier without a verified domain, emails can only be sent to your Resend account email. Verify a custom domain at resend.com/domains to send to any email.

Architecture

• Frontend: Next.js app with real-time updates via Supabase Realtime
• Database: PostgreSQL (Supabase) with pg_cron for scheduling and pgvector for semantic search
• Authentication: Supabase Auth (Email/Password + Google OAuth)
• AI Agent: OpenAI GPT-4 with function calling (search & scrape tools)
• AI Summaries: Auto-generated one-sentence summaries with vector embeddings for each successful execution
• Edge Function: Deno-based serverless function that orchestrates agent execution
• Web Scraping: Firecrawl API for search and content extraction (supports per-user API keys via partner integration)

Scalable Dispatcher Architecture

Open Scouts uses a dispatcher pattern designed to scale to thousands of scouts:

Every minute:
pg_cron → dispatch_due_scouts() → finds due scouts → pg_net HTTP POST
                                                          ↓
                                    ┌──────────────────────┼──────────────────────┐
                                    ↓                      ↓                      ↓
                              Edge Function          Edge Function          Edge Function
                              (scout A)              (scout B)              (scout C)
                              [isolated]             [isolated]             [isolated]

• Dispatcher (SQL): Runs every minute via pg_cron, queries for due scouts, and fires individual HTTP requests
• Isolated Execution: Each scout runs in its own edge function invocation with full resources (256MB memory, 400s timeout)
• Automatic Cleanup: A separate cron job cleans up stuck executions every 5 minutes
• Vault Integration: Supabase credentials are securely stored in the vault and read by the dispatcher

Security

• Row Level Security (RLS): All database tables have RLS policies ensuring users can only access their own data
• User Isolation: Scouts, messages, and executions are all tied to authenticated users
• Secure Auth Flow: OAuth tokens and sessions are managed by Supabase Auth
• Service Role: Server-side operations (cron jobs, edge functions) use service role for privileged access
• API Key Storage: Firecrawl API keys (when using partner integration) are stored server-side in user_preferences and never exposed to the client

Build for Production

bun run build  # or: npm run build / pnpm run build
bun start      # or: npm start / pnpm start

Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

License

MIT