Beyond chat sessions: the Pareto frontier for context.
Memory benchmarks tested in December 2025. Alchemyst serves context at 170ms P50 latency for $0.06 per 1M tokens, with a memory F1 of 0.76. The charts below show what that tradeoff looks like against Supermemory, Zep, and Hindsight GPT.
Tested December 2025 · Prices in USD · Performance is F1
- P50 latency
- 170ms
- per 1M tokens
- $0.06
- value ratio
- 12x
- memory F1
- 0.76
Alchemyst AI memory benchmarks, December 2025
Overall: Alchemyst serves context at 170ms P50 latency for $0.06 per 1M tokens, with 0.76 memory F1. Hindsight GPT OSS 120B scores 0.907 F1 at $0.36 with 450ms latency. Supermemory scores 0.833 F1 at $6.33 with 820ms latency. Zep scores 0.723 F1 at $12.50 with 1150ms latency. Prices are USD in the test harness. Performance is F1.
Value index (performance per dollar)
Alchemyst delivers over 12x more performance value per dollar than the nearest competitor, and costs 83 percent less than Zep. The gap comes from cost, not from giving up accuracy.
Per-category F1 results
- Single Session Preference (n=30): Hindsight GPT 0.867, Supermemory 0.700, Alchemyst 0.600, Zep 0.567.
- Single Session User (n=68): Hindsight GPT 1.000, Supermemory 0.971, Alchemyst 0.956, Zep 0.929.
- Knowledge Update (n=75): Hindsight GPT 0.923, Supermemory 0.884, Zep 0.833, Alchemyst 0.560.
- Single Session Assistant (n=55): Hindsight GPT 0.982, Alchemyst 0.964, Supermemory 0.964, Zep 0.804.
- Temporal Reasoning (n=131): Hindsight GPT 0.857, Supermemory 0.767, Alchemyst 0.756, Zep 0.624.
- Multi Session (n=133): Hindsight GPT 0.812, Alchemyst 0.729, Supermemory 0.714, Zep 0.579.
Methodology
Evals cover single session preference, user, and assistant recall, knowledge updates, temporal reasoning, and multi session memory, with 30 to 133 cases per category. Latency is P50 for a retrieval call in the test harness. Reproduce the setup against your own workload before making a buying decision. Full per-category tables are in the data archive section of https://getalchemystai.com/benchmarks.
Intelligence delivered per dollar.
Performance divided by cost. Alchemyst delivers more than 12x the performance value per dollar of the nearest competitor because the price denominator is an order of magnitude smaller.
Alchemyst delivers over 12x more performance value per dollar compared to the nearest competitor.
Reading the chart
Alchemyst delivers over 12x more performance value per dollar compared to the nearest competitor.
- Cost savings vs Zep
- 83% lower
- Latency, P50
- 170ms
- Overall memory F1
- 0.76
The gap comes from cost, not from giving up accuracy. Alchemyst stays competitive on F1 while indexing and serving at a fraction of the price.
Cost on a log scale, performance on a linear one.
Cheaper is to the left, better is to the top. Alchemyst sits alone in the low cost, high performance corner. The line connects providers sorted by price to show the frontier.
Head to head across six memory dimensions.
One row per eval category. Alchemyst leads on efficiency everywhere and stays within striking distance on raw F1, including 0.956 on Single Session User and 0.964 on Single Session Assistant.
| Dimension | Alchemyst | Supermemory | Zep | Hindsight GPT |
|---|---|---|---|---|
| Single Session Preference (n=30) | 0.600 | 0.700 | 0.567 | 0.867 |
| Single Session User (n=68) | 0.956 | 0.971 | 0.929 | 1.000 |
| Knowledge Update (n=75) | 0.560 | 0.884 | 0.833 | 0.923 |
| Single Session Assistant (n=55) | 0.964 | 0.964 | 0.804 | 0.982 |
| Temporal Reasoning (n=131) | 0.756 | 0.767 | 0.624 | 0.857 |
| Multi Session (n=133) | 0.729 | 0.714 | 0.579 | 0.812 |
Every category, every provider.
Raw per-category F1, sorted by performance within each table. Prices are USD per 1K in the test harness. This is the complete dataset behind the charts above.
Single Session Preference
n=30| Model | Price (USD/1K) | Performance (F1) |
|---|---|---|
| Hindsight GPT OSS 120B | $0.3600 | 0.867 |
| Supermemory | $6.3330 | 0.700 |
| Alchemyst | $0.0600 | 0.600 |
| Zep | $12.5000 | 0.567 |
Single Session User
n=68| Model | Price (USD/1K) | Performance (F1) |
|---|---|---|
| Hindsight GPT | $0.3600 | 1.000 |
| Supermemory | $6.3330 | 0.971 |
| Alchemyst | $0.0600 | 0.956 |
| Zep | $12.5000 | 0.929 |
Knowledge Update
n=75| Model | Price (USD/1K) | Performance (F1) |
|---|---|---|
| Hindsight GPT | $0.3600 | 0.923 |
| Supermemory | $6.3330 | 0.884 |
| Zep | $12.5000 | 0.833 |
| Alchemyst | $0.0600 | 0.560 |
Single Session Assistant
n=55| Model | Price (USD/1K) | Performance (F1) |
|---|---|---|
| Hindsight GPT | $0.3600 | 0.982 |
| Supermemory | $6.3330 | 0.964 |
| Alchemyst | $0.0600 | 0.964 |
| Zep | $12.5000 | 0.804 |
Temporal Reasoning
n=131| Model | Price (USD/1K) | Performance (F1) |
|---|---|---|
| Hindsight GPT | $0.3600 | 0.857 |
| Supermemory | $6.3330 | 0.767 |
| Alchemyst | $0.0600 | 0.756 |
| Zep | $12.5000 | 0.624 |
Multi Session
n=133| Model | Price (USD/1K) | Performance (F1) |
|---|---|---|
| Hindsight GPT | $0.3600 | 0.812 |
| Alchemyst | $0.0600 | 0.729 |
| Supermemory | $6.3330 | 0.714 |
| Zep | $12.5000 | 0.579 |
Methodology note
Evals cover single session preference, user, and assistant recall, knowledge updates, temporal reasoning, and multi session memory. Counts range from 30 to 133 cases per category. Latency is P50 for a retrieval call in the test harness. Reproduce the setup against your own workload before making a buying decision, and contact the team for the eval definitions.