stockpile

An enrichment cache for GTM data

---

What it does

Stockpile is a self-hostable HTTP cache for enrichment APIs like Apollo, Hunter, Findymail, Icypeas, and Clearbit. It is built for Clay tables and other no-code workflows that can make JSON HTTP requests.

Clay row
  -> POST /lookup to Stockpile
    -> cache hit: use the cached result
    -> cache miss: call the enrichment provider
      -> POST /store to Stockpile
        -> future rows reuse the saved result

Every cache slot is keyed by:

provider x endpoint x normalized inputs

That means Apollo people-match, Apollo phone-reveal, and Hunter email-finder can all store separate values for the same lead. Stockpile never guesses that two different input shapes describe the same person. It only matches exact normalized inputs, because a wrong cache hit is worse than spending a credit.

Getting started

Step 1: Deploy Stockpile

Click the Railway button:

Railway step-by-step

1. Click the Deploy on Railway button above

2. Sign in to Railway (or create an account)

3. If asked to name the project, pick anything you like, such as stockpile

4. Leave the generated variables as-is. Railway creates an API_KEY for you

5. Deploy the template

6. Wait for the build to finish

7. Open the stockpile service and copy the public Railway URL, such as:

   https://stockpile-production-xxxx.up.railway.app
   

8. Open the service variables and copy API_KEY

The template creates one service, one public domain, one generated API_KEY, and one /data volume for SQLite persistence.

Local / VPS / other

git clone https://github.com/eliasstravik/stockpile.git
cd stockpile
npm install
cp .env.example .env
npm run build
npm start

For production outside Railway, set:

API_KEY=your-long-random-token
DB_PATH=/data/cache.db
PORT=3000

Your server URL is whatever domain or IP points to this process.

Step 2: Save your Stockpile URL and API key

You will use these in Clay:

cache_url = https://your-stockpile-url
cache_api_key = the API_KEY from Railway variables

Every authenticated request uses:

Authorization: Bearer YOUR_API_KEY
Content-Type: application/json

Step 3: Add a cache_check column in Clay

Add an HTTP API column before your enrichment provider column.

Method: POST

URL:

{{cache_url}}/lookup

Headers:

• Key: Authorization
• Value: Bearer {{cache_api_key}}
• Key: Content-Type
• Value: application/json

Turn on Clay's treat 404 as success option. Stockpile intentionally returns 404 for a single-slot miss so your row can branch cleanly.

Body:

{
  "provider": "apollo",
  "endpoint": "people-match",
  "inputs": {
    "first_name": "{{First Name}}",
    "last_name": "{{Last Name}}",
    "company_domain": "{{Company Domain}}"
  }
}

If there is a hit, Stockpile returns:

{
  "hit": true,
  "results": [
    {
      "provider": "apollo",
      "endpoint": "people-match",
      "value": {
        "email": "john@acme.com",
        "title": "VP Sales"
      },
      "created_at": 1718654400000,
      "expires_at": null
    }
  ]
}

Step 4: Run the enrichment only on a miss

Add your actual Apollo, Hunter, Findymail, Icypeas, or other provider column after cache_check.

Use this run condition:

{{cache_check.hit}} != true

Step 5: Store successful enrichments

Add another HTTP API column after the provider call.

Method: POST

URL:

{{cache_url}}/store

Headers:

• Key: Authorization
• Value: Bearer {{cache_api_key}}
• Key: Content-Type
• Value: application/json

Body:

{
  "provider": "apollo",
  "endpoint": "people-match",
  "inputs": {
    "first_name": "{{First Name}}",
    "last_name": "{{Last Name}}",
    "company_domain": "{{Company Domain}}"
  },
  "value": {
    "email": "{{enrichment_call.email}}",
    "title": "{{enrichment_call.title}}"
  },
  "ttl": 2592000
}

Use this run condition:

{{enrichment_call}} != null

Store the small normalized object you want to reuse later, not necessarily the provider's entire raw response. For example, storing { "work_email": "{{enrichment_call.email}}" } makes downstream Clay formulas easier.

Step 6: Use the result

Add a formula column that chooses the cached value when present, otherwise the live enrichment result:

{{cache_check.results[0].value}} OR {{enrichment_call}}

For a specific field:

{{cache_check.results[0].value.work_email}}

Downstream columns should read from this result column instead of reading directly from the provider column.

That's it

From now on, rows check Stockpile first. Hits reuse the cached value. Misses call the provider once, then save the result for the next matching lookup.

Common lookup patterns

Search one provider and endpoint:

{
  "provider": "apollo",
  "endpoint": "people-match",
  "inputs": { "email": "{{Email}}" }
}

Search across providers for one endpoint:

{
  "endpoint": "email-finder",
  "inputs": { "email": "{{Email}}" },
  "sort": { "providers": ["icypeas", "hunter", "apollo"] }
}

Get everything one provider has for a lead:

{
  "provider": "apollo",
  "inputs": { "email": "{{Email}}" }
}

Search a bounded provider and endpoint set:

{
  "providers": ["apollo", "hunter"],
  "endpoints": ["people-match", "email-finder"],
  "inputs": { "email": "{{Email}}" }
}

Default sort order is requested provider order, requested endpoint order, newest stored result first, then provider and endpoint alphabetically. Override it with:

{
  "sort": {
    "providers": ["icypeas", "hunter", "apollo"],
    "endpoints": ["find-work-email", "email-finder"],
    "created_at": "desc"
  }
}

API reference

All endpoints return JSON. All endpoints except GET /health require bearer token auth.

GET /health

Railway health check. No auth.

curl https://your-stockpile-url/health

{ "ok": true, "count": 0 }

GET /stats

Authenticated row count and database path.

curl https://your-stockpile-url/stats \
  -H "Authorization: Bearer $API_KEY"

{ "count": 12, "db_path": "/data/cache.db" }

POST /lookup

Looks up cached values for one normalized inputs object. inputs is required. Provider and endpoint filters are optional and combine with AND.

curl -X POST https://your-stockpile-url/lookup \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "provider":"apollo",
    "endpoint":"people-match",
    "inputs":{"email":"john@acme.com"}
  }'

Single-slot hits return 200. Single-slot misses return 404. Multi-result lookups always return 200.

{
  "hit": true,
  "results": [
    {
      "provider": "apollo",
      "endpoint": "people-match",
      "value": { "email": "john@acme.com" },
      "created_at": 1718654400000,
      "expires_at": null
    }
  ]
}

POST /store

Stores or overwrites one cache slot. ttl is seconds. Omit it or pass 0 for no expiry.

curl -X POST https://your-stockpile-url/store \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "provider":"apollo",
    "endpoint":"people-match",
    "inputs":{"email":"john@acme.com"},
    "value":{"email":"john@acme.com","title":"VP Sales"},
    "ttl":2592000
  }'

{
  "ok": true,
  "provider": "apollo",
  "endpoint": "people-match",
  "inputs_hash": "9e8d6e9d4b1d2b7f72a3d6f3c32d3a31",
  "expires_at": 1718654400000
}

POST /invalidate

Deletes rows using the same filter semantics as lookup. At least one filter is required.

curl -X POST https://your-stockpile-url/invalidate \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"provider":"apollo","endpoint":"people-match","inputs":{"email":"john@acme.com"}}'

Other useful invalidations:

{ "endpoint": "email-finder", "inputs": { "email": "john@acme.com" } }
{ "provider": "apollo", "inputs": { "email": "john@acme.com" } }
{ "inputs": { "email": "john@acme.com" } }

To delete the entire cache, use an explicit confirmation body:

{ "all": true, "confirm": "delete_all" }

POST /debug-key

Shows exactly how Stockpile normalizes provider, endpoint, and inputs before hashing.

curl -X POST https://your-stockpile-url/debug-key \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"provider":"Apollo","endpoint":"People Match","inputs":{"Email":"JOHN@Acme.com "}}'

{
  "canonical": { "email": "john@acme.com" },
  "provider": "apollo",
  "endpoint": "people-match",
  "inputs_hash": "..."
}

Provider names

Suggested provider and endpoint names:

Provider	Endpoint	What it caches
apollo	people-match	Apollo /people/match person enrichment
apollo	phone-reveal	Apollo phone reveal
apollo	org-enrich	Apollo /organizations/enrich
hunter	email-finder	Hunter email finder
hunter	email-verify	Hunter email verification
findymail	email-finder	Findymail email lookup
icypeas	find-work-email	Icypeas work email lookup

Provider and endpoint tags are normalized by trimming, lowercasing, and replacing whitespace with dashes. "People Match" becomes people-match.

How normalization works

Inputs are normalized before hashing so harmless formatting differences still hit the same cache slot.

String rules:

1. Trim whitespace and lowercase
2. If it starts with http:// or https://, or looks like a domain, strip the scheme, leading www., query, hash, and trailing slashes
3. Else if it is phone-like and contains at least seven digits, strip all non-digits
4. Else keep the trimmed lowercase string

Object rules:

• Drop null, undefined, and empty strings
• Lowercase object keys and sort them before hashing
• Normalize arrays recursively while preserving order
• Keep numbers and booleans as-is

The normalized value is JSON serialized, sha256 hashed, and truncated to the first 32 hex characters.

Environment variables

Variable	Required	Default	Description
API_KEY	Yes for auth		Bearer token for authenticated endpoints
DB_PATH	No	/data/cache.db	SQLite database path
PORT	No	3000	Server port
ALLOW_UNAUTHENTICATED	No		Set to true only for local development without auth
MAX_REQUEST_BYTES	No	1048576	Maximum JSON request body size
MAX_VALUE_BYTES	No	262144	Maximum serialized value size for /store

Stockpile refuses to start without API_KEY unless ALLOW_UNAUTHENTICATED=true is set. The Railway template generates API_KEY automatically.

Development

npm install
cp .env.example .env
npm run dev

Command	Description
npm run dev	Start dev server with hot reload
npm run build	Type-check and build to dist/
npm start	Run the production build
npm run typecheck	Type-check without writing dist/
npm run test:normalize	Run normalization tests
npm run test:integration	Run HTTP integration tests
npm run test:production	Smoke-test the compiled production server
npm test	Run normalization and integration tests

Architecture

Built with Hono, TypeScript, and SQLite through better-sqlite3. It is one Node process and one SQLite file.

src/
  index.ts       HTTP server, routes, SQLite setup
  normalize.ts   input canonicalization and hashing

Railway mounts a persistent volume at /data, so the production database lives at /data/cache.db.

Security measures

• Bearer token auth on all endpoints except /health
• Exact-match cache keys after deterministic normalization
• No fuzzy matching or provider-specific identity guessing
• Expired rows are deleted on read and by a background sweep
• No external database service or third-party cache service required

License

MIT