Usage Patterns Primer

// ❌ Flat structure - Hard to query effectively
{
  "groupName": ["marketing_q4_2024_campaign_email_newsletter_product_launch"]
}

// ✅ Hierarchical structure - Composable and flexible
{
  "groupName": ["marketing", "q4_2024", "campaign", "email"]
}

// Base document - accessible to all engineering
{
  "groupName": ["engineering"],
  "content": "Company-wide coding standards..."
}

// Backend auth code - specific team access
{
  "groupName": ["engineering", "backend", "auth"],
  "content": "JWT middleware implementation..."
}

// Frontend component - specific project
{
  "groupName": ["engineering", "frontend", "project_redesign"],
  "content": "Button component with new design system..."
}

// Broad: All engineering docs (might return 10,000+ docs)
search({ groupName: ["engineering"] })

// Focused: Backend authentication (returns ~50 docs)
search({ groupName: ["engineering", "backend", "auth"] })

// Specific: Auth docs in current project (returns ~10 docs)
search({
  groupName: ["engineering", "backend", "auth"],
  metadata: { project: "api_v2" }
})

// Document has changed - replace it entirely
async function updateDocument(fileName, newContent) {
  // Step 1: Find and delete old version
  const existing = await alchemyst.search({
    metadata: { fileName: fileName }
  });

  if (existing.results.length > 0) {
    await alchemyst.delete(existing.results[0].id);
  }

  // Step 2: Add new version
  await alchemyst.add({
    content: newContent,
    metadata: { fileName: fileName }
  });
}

// Use case: Documentation that gets edited
// "user-guide.pdf" updated from v1 to v2

// Track document evolution over time
await alchemyst.add({
  content: quarterlyReport,
  metadata: {
    fileName: "q4-report-v1.pdf",
    version: "1.0",
    status: "draft"
  }
});

await alchemyst.add({
  content: quarterlyReportFinal,
  metadata: {
    fileName: "q4-report-v2.pdf",  // Different fileName!
    version: "2.0",
    status: "final",
    supersedes: "q4-report-v1.pdf"
  }
});

// Use case: Legal documents, contracts, specs where history matters

// For chat-like interactions where context accumulates
await alchemyst.addToConversation({
  conversationId: "support-ticket-1234",
  content: "Customer reported login issue...",
  metadata: {
    timestamp: "2024-12-29T10:00:00Z",
    speaker: "customer"
  }
});

await alchemyst.addToConversation({
  conversationId: "support-ticket-1234",
  content: "Resolved by resetting password...",
  metadata: {
    timestamp: "2024-12-29T10:15:00Z",
    speaker: "agent"
  }
});

// Use case: Support tickets, chat logs, ongoing projects

Is the new content completely replacing the old?
├─ YES → Use Strategy A (Delete-Then-Add)
│
├─ NO → Does history matter?
    ├─ YES → Use Strategy B (Versioned FileNames)
    │
    └─ NO → Is this part of a conversation/thread?
        ├─ YES → Use Strategy C (Conversational)
        └─ NO → Use Strategy B with cleanup job

// ❌ Don't do this - 100 sequential operations
for (const doc of documents) {
  await alchemyst.delete(doc.id);
  await alchemyst.add(doc.updated);
}
// Time: ~30-60 seconds

// ✅ Do this - bulk operations
const deleteIds = documents.map(d => d.id);
await alchemyst.bulkDelete(deleteIds);

await alchemyst.bulkAdd(
  documents.map(d => d.updated)
);
// Time: ~3-5 seconds

// ✅ Good - store together
{
  groupName: ["product", "api_spec"],
  content: `
    Authentication Endpoint
    POST /api/auth/login

    Request Body:
    - email: string (required)
    - password: string (required)

    Response:
    - token: JWT string
    - expiresIn: number (seconds)

    Error Codes:
    - 401: Invalid credentials
    - 429: Rate limited
  `
}

// ❌ Bad - unnecessarily fragmented
// Document 1: "POST /api/auth/login"
// Document 2: "Request body fields"
// Document 3: "Response format"
// Document 4: "Error codes"
// Problem: User query "how to login" needs all 4 docs

// If users always need A, B, and C together, combine them
// Example: Product onboarding guide (overview + setup + first steps)

// ✅ Good - separate by access control
// Public documentation
{
  groupName: ["docs", "public"],
  content: "How to use our API..."
}

// Internal implementation notes
{
  groupName: ["docs", "internal"],
  content: "Rate limiting implementation details..."
}

// ✅ Good - separate by volatility
// Static content (rarely changes)
{
  fileName: "company-history.md",
  content: "Founded in 2020..."
}

// Dynamic content (changes daily)
{
  fileName: "daily-metrics-2024-12-29.json",
  content: "Today's sales: $45,231..."
}

// One 50,000-word document:
// ❌ Slow to retrieve
// ❌ Wastes tokens on irrelevant sections
// ❌ Hard to rank relevance

// Split into 10 logical sections:
// ✅ Fast retrieval
// ✅ Only relevant sections retrieved
// ✅ Better semantic ranking

// ❌ Too small (100 words)
// Problem: Loses context, requires many docs for complete answer
{
  content: "JWT tokens expire after 24 hours"
}

// ❌ Too large (10,000 words)
// Problem: Retrieves too much irrelevant content
{
  content: "Entire authentication system documentation..."
}

// ✅ Just right (800 words)
// Problem: Single cohesive topic with enough context
{
  content: `
    JWT Token Management

    Overview: Our JWT implementation uses...

    Token Lifecycle:
    1. Generation: POST /auth/login
    2. Validation: Automatic on each request
    3. Refresh: POST /auth/refresh
    4. Expiration: 24 hours default

    [Full details with examples...]
  `
}

// Document with references to related content
{
  groupName: ["engineering", "backend", "auth"],
  metadata: {
    relatedDocs: ["api-security-guide", "user-model-schema"],
    prerequisites: ["basic-auth-concepts"]
  },
  content: "Advanced authentication patterns..."
}

// Your application can fetch related docs for deeper context

// Sequential adds: ~30 seconds for 1000 docs
for (let i = 0; i < 1000; i++) {
  await alchemyst.add(documents[i]);
}

// Async add: ~3 seconds for 1000 docs (10x faster!)
// documents is an array of content
const documents = [];
documents.push({
  content: "files content",
  metadata: { // optional
    file_name: "file_name",
    file_type: "pdf/txt/json",
    group_name: ["group1", "group2"],
  },
});

await alchemyst.add(documents);

// Process 10,000 documents efficiently
async function bulkIngest(documents) {
  const BATCH_SIZE = 1000;
  const batches = [];

  // Split into batches of 1000
  for (let i = 0; i < documents.length; i += BATCH_SIZE) {
    batches.push(documents.slice(i, i + BATCH_SIZE));
  }

  // Process batches with progress tracking
  const results = [];
  for (let i = 0; i < batches.length; i++) {
    console.log(`Processing batch ${i + 1}/${batches.length}`);

    const result = await alchemyst.bulkAdd(batches[i]);
    results.push(result);

    // Optional: Small delay between batches to avoid rate limits
    if (i < batches.length - 1) {
      await new Promise(resolve => setTimeout(resolve, 100));
    }
  }

  return results;
}

// Process multiple batches in parallel (but not too many!)
async function parallelBulkIngest(documents) {
  const BATCH_SIZE = 1000;
  const MAX_CONCURRENT = 3; // Don't overwhelm the API

  const batches = [];
  for (let i = 0; i < documents.length; i += BATCH_SIZE) {
    batches.push(documents.slice(i, i + BATCH_SIZE));
  }

  // Process 3 batches at a time
  const results = [];
  for (let i = 0; i < batches.length; i += MAX_CONCURRENT) {
    const chunk = batches.slice(i, i + MAX_CONCURRENT);
    const chunkResults = await Promise.all(
      chunk.map(batch => alchemyst.bulkAdd(batch))
    );
    results.push(...chunkResults);

    console.log(`Completed ${Math.min(i + MAX_CONCURRENT, batches.length)}/${batches.length} batches`);
  }

  return results;
}

async function robustBulkIngest(documents) {
  const BATCH_SIZE = 1000;
  const MAX_RETRIES = 3;

  const batches = chunkArray(documents, BATCH_SIZE);
  const results = [];
  const failures = [];

  for (let i = 0; i < batches.length; i++) {
    let attempt = 0;
    let success = false;

    while (attempt < MAX_RETRIES && !success) {
      try {
        const result = await alchemyst.bulkAdd(batches[i]);
        results.push(result);
        success = true;
      } catch (error) {
        attempt++;
        console.warn(`Batch ${i} failed (attempt ${attempt}/${MAX_RETRIES}):`, error.message);

        if (attempt < MAX_RETRIES) {
          // Exponential backoff: 1s, 2s, 4s
          await new Promise(resolve =>
            setTimeout(resolve, 1000 * Math.pow(2, attempt - 1))
          );
        } else {
          // Final failure - log and continue
          failures.push({ batchIndex: i, error: error.message });
        }
      }
    }
  }

  if (failures.length > 0) {
    console.error(`${failures.length} batches failed after retries:`, failures);
  }

  return { results, failures };
}

// For 100K+ documents, give users feedback
async function ingestWithProgress(documents, onProgress) {
  const BATCH_SIZE = 1000;
  const batches = chunkArray(documents, BATCH_SIZE);

  let completed = 0;

  for (const batch of batches) {
    await alchemyst.bulkAdd(batch);
    completed += batch.length;

    onProgress({
      completed,
      total: documents.length,
      percentage: (completed / documents.length * 100).toFixed(1)
    });
  }
}

// Usage in UI
await ingestWithProgress(allDocs, (progress) => {
  console.log(`Progress: ${progress.percentage}% (${progress.completed}/${progress.total})`);
  // Update progress bar, etc.
});

// ❌ Anti-pattern: One document per field
// Ticket #1234 split into 6 documents:
{
  groupName: ["support", "ticket_1234", "subject"],
  content: "Cannot login to dashboard"
}
{
  groupName: ["support", "ticket_1234", "description"],
  content: "User reports error message..."
}
{
  groupName: ["support", "ticket_1234", "priority"],
  content: "High"
}
{
  groupName: ["support", "ticket_1234", "agent"],
  content: "John Doe"
}
{
  groupName: ["support", "ticket_1234", "resolution"],
  content: "Password reset required"
}
{
  groupName: ["support", "ticket_1234", "timestamp"],
  content: "2024-12-29T10:00:00Z"
}

// Problems:
// 1. Query "login issues" retrieves 6 fragments
// 2. LLM needs to reassemble the ticket
// 3. Costs 6x the storage and retrieval
// 4. Loses contextual relationships

// ✅ Correct: One cohesive document
{
  groupName: ["support", "tickets", "resolved"],
  metadata: {
    ticketId: "1234",
    priority: "high",
    agent: "John Doe",
    resolvedAt: "2024-12-29T10:15:00Z"
  },
  content: `
    Ticket #1234: Cannot login to dashboard

    Description:
    User reports error message "Invalid credentials" when attempting
    to log in with correct password. Account verified as active.

    Resolution:
    Issue caused by expired password (90-day policy). Password reset
    link sent to user. User confirmed successful login after reset.

    Root Cause: Password expiration notification system failed to
    send warning emails.
  `
}

// Benefits:
// 1. Single retrieval gets complete context
// 2. LLM can understand full story
// 3. 6x cost reduction
// 4. Preserves semantic relationships

// User query: "How do we handle login issues?"

// With over-segmentation:
// → Retrieves 47 fragments across 8 tickets
// → LLM confused by incomplete pieces
// → Answer: Generic, not helpful

// With proper documents:
// → Retrieves 3-4 complete ticket resolutions
// → LLM sees full context and patterns
// → Answer: "Based on tickets #1234, #1456, we typically..."

// This is OK even though it's short
{
  groupName: ["company", "contacts"],
  metadata: { department: "engineering", role: "cto" },
  content: "Jane Smith - CTO - jane@company.com - ext. 1234"
}

// ❌ Metadata bloat
{
  groupName: ["products", "electronics"],
  metadata: {
    productId: "PROD-12345",
    productName: "Wireless Mouse",
    productDescription: "Ergonomic wireless mouse with 6 buttons",
    productCategory: "Electronics",
    productSubcategory: "Computer Accessories",
    productSubSubcategory: "Input Devices",
    productBrand: "TechCorp",
    productBrandId: "BRAND-789",
    productBrandCountry: "USA",
    productPrice: 29.99,
    productPriceCurrency: "USD",
    productPriceFormatted: "$29.99",
    productStock: 150,
    productStockStatus: "in_stock",
    productStockWarehouse: "warehouse_3",
    productWeight: "0.2kg",
    productDimensions: "12x8x4cm",
    productColor: "black",
    productColorHex: "#000000",
    productRating: 4.5,
    productReviewCount: 234,
    productSKU: "MOUSE-WL-BK-001",
    productUPC: "123456789012",
    productManufacturer: "TechCorp Electronics Ltd",
    productOriginCountry: "China",
    productWarranty: "2 years",
    productReleaseDate: "2024-01-15",
    productLastUpdated: "2024-12-29T10:00:00Z",
    productCreatedAt: "2024-01-01T00:00:00Z",
    productCreatedBy: "admin@company.com",
    productUpdatedBy: "inventory@company.com",
    productTags: ["wireless", "ergonomic", "6-button", "usb-receiver"],
    productIsActive: true,
    productIsFeatured: false,
    productIsDiscounted: false
    // ... 15 more fields
  },
  content: "TechCorp Wireless Mouse - Ergonomic design with 6 programmable buttons..."
}

// Problems:
// 1. 90% of metadata never used in queries
// 2. Duplicates info already in content
// 3. Slows down indexing and retrieval
// 4. Harder to maintain consistency

// ✅ Lean metadata - only what you query/filter by
{
  groupName: ["products", "electronics", "accessories"],
  metadata: {
    productId: "PROD-12345",
    category: "input_devices",
    price: 29.99,
    inStock: true,
    brand: "TechCorp"
  },
  content: `
    TechCorp Wireless Mouse (SKU: MOUSE-WL-BK-001)

    Ergonomic wireless mouse with 6 programmable buttons
    Price: $29.99 | Rating: 4.5/5 (234 reviews)
    Color: Black | Weight: 0.2kg | Dimensions: 12x8x4cm

    Features:
    - 2.4GHz wireless with USB receiver
    - Ergonomic design for all-day comfort
    - 6 programmable buttons
    - 2-year warranty

    Stock: 150 units available at Warehouse 3
    Origin: Made in China | Released: January 2024
  `
}

// Benefits:
// 1. 80% less metadata storage
// 2. Faster queries (fewer fields to scan)
// 3. Easier to maintain
// 4. Full info still retrievable from content

Choose Pattern 1 (Hierarchical groupName) when:
→ You have multiple teams/departments
→ Access control matters
→ Your data has natural categories

Choose Pattern 2 (Document Updates) when:
→ Delete-then-add: Content genuinely changes
→ Versioning: History matters (legal, compliance)
→ Conversational: Building context over time

Choose Pattern 3 (Composable Context) when:
→ Combine: Info always retrieved together
→ Split: Different access patterns or update frequencies
→ Target 500-2000 words per document

Choose Pattern 4 (Bulk Operations) when:
→ Ingesting 1000+ documents
→ Batch size: 1000 documents per call
→ Use retries and progress tracking

Avoid Anti-Pattern 1 (Over-segmentation) by:
→ Keeping related info together
→ Passing the "paragraph test"
→ Simulating real user queries

Avoid Anti-Pattern 2 (Metadata Bloat) by:
→ Starting with 5 fields max
→ Only storing what you filter/sort by
→ Keeping descriptions in content