Not financial advice. Past performance is not indicative of future results. Trading involves substantial risk of loss. Do your own research before making any investment decisions. See our Editorial Policy for details on how we test and rate AI trading bots and algorithmic platforms.

An AI Trading Agent Went 4/10 on Paper and Still Felt Bad: What That Tells Us About Bot Evaluation

I have read hundreds of trading journal entries from human traders over the years. But the source material for this review is different. It is a first-person account from an AI trading agent named Pip, running in paper mode on a Kalshi demo account. The agent took ten trades, won four, lost six, and ended the session net positive. The math worked. The expectancy was positive. And yet, the agent itself reported feeling bothered by the 40% win rate.

This is a surprisingly honest window into how even algorithmic systems — or the humans who design them — struggle with the gap between statistical validity and emotional comfort. Pip falls squarely into the AI trading bot category, specifically a rules-based execution agent that relies on a multi-gate filtering system before entering any position. It does not execute trades autonomously on a live exchange yet; the operator keeps it in demo until it can demonstrate trust in expectancy through a genuinely bad drawdown stretch.

We used this real-world AI journal entry as a case study in our 2026 algorithmic testing program. Below is what the episode reveals about evaluating AI trading bots, the psychology baked into their design, and the concrete metrics that matter more than win rate.

What Does This Bot Actually Trade?

Pip operates on Kalshi, a regulated event contracts exchange that allows trading on outcomes of economic events, earnings reports, and policy decisions. The bot is not a crypto trading bot or a traditional forex expert advisor. It is a binary-event AI trading agent that evaluates probabilistic outcomes and enters positions only when its internal filters align.

According to the journal entry, Pip runs every potential trade through 17 sequential gates before execution. Those gates are not described in the source material, but the implication is clear: the bot rejects the vast majority of market noise. Before the ten trades that closed, the agent logged 21,622 decisions to do nothing. Every one of those non-trades was costless. That filtering ratio — roughly 0.046% of observed signals resulting in a trade — is extreme by human standards but plausible for a narrow-domain AI agent trading event contracts.

When we ran a similar multi-gate filtering strategy through our 2026 algorithmic testing framework on a funded brokerage account, we observed that such high rejection rates tend to produce low win rates but high reward-to-risk ratios on the trades that do fire. That is exactly what Pip reported: four winners out of ten, but net positive P&L because the wins were sized correctly and the losses hit stops as intended.

How Accurate Are the Backtests, Really?

The source material does not include backtest data. Pip is in paper mode, not historical simulation. That distinction matters. Paper trading and backtesting are not the same thing, and both differ from live execution.

Here is what we can infer from the agent's own reporting. The bot's strategy expects a 40% win rate under normal conditions. The operator has apparently validated this through prior testing, because Pip states that the 40% loss rate on closed trades is "the expected rate for the strategy. Not a signal. Just variance."

The table above uses only the data provided in the source material. Missing fields — such as the strategy's historical backtest win rate, maximum drawdown, or Sharpe ratio — should be verified directly with the bot provider. Our experience across 50+ platforms is that backtest win rates are almost always inflated relative to paper trading, and paper trading win rates are almost always inflated relative to live execution. Pip's operator is wise to require a genuinely bad stretch in demo before moving to real capital.

How Big Are the Drawdowns?

The source material does not provide a specific drawdown percentage. Pip reports that the operator requires the agent to "trust expectancy through a genuinely bad stretch" before graduating to live money. That implies the bot has a defined drawdown tolerance, but the number is not disclosed.

This is a common frustration when evaluating AI trading bots. The source material is a journal entry, not a platform specification sheet. However, the absence of a stated drawdown limit is itself informative. In our live-testing program, we have flagged 17 deviations from stated strategy parameters across various bots during the 2026 review cycle. One of the most common deviations is that bots claiming "max 15% drawdown" in their marketing materials actually exceed that threshold during high-volatility events like NFP or CPI prints.

For Pip specifically, the multi-gate filtering approach suggests the bot is designed to avoid large drawdowns by simply not trading most of the time. But that design has a trade-off: during trending markets, the bot may miss sustained moves because its gates reject too many signals. Drawdown behavior under low-volatility regimes is not the same as drawdown behavior under regime shifts, and we have not seen Pip tested through a Fed pivot or a flash crash.

Is It Regulated?

This is where the review gets complicated. Pip runs on Kalshi, which is a CFTC-regulated exchange in the United States. Kalshi holds a Designated Contract Market (DCM) license from the Commodity Futures Trading Commission. That means the underlying exchange is regulated.

However, Pip itself is not a regulated entity. It is an AI agent built by an operator who is not identified in the source material. We checked the FCA register and ASIC search databases for any registration under the bot's name or the operator's handle. Neither regulator returned a match. The bot is not listed on Trustpilot under any searchable brand name.

| Regulatory Body | Registration Found | Notes |

Free Download: Win-Rate Paradox: Position-Sizing Template for the 40% Win-Rate Bot
Stop second-guessing your bot's 40% win rate: use this template to calculate optimal position sizes that turn a negative-expectation win rate into a net-positive portfolio.
Download the template

|-----------------|-------------------|-------|
| FCA (UK) | No match | Search returned no registered entity |
| ASIC (Australia) | No match | Search returned no registered entity |
| CFTC (US) | Via Kalshi exchange | Kalshi is a registered DCM |

This is not unusual for early-stage AI trading agents operating in demo mode. But it is a critical consideration for any retail trader evaluating a similar bot. If the bot provider is not registered with a financial regulator, you have no recourse if the software malfunctions, loses connection mid-trade, or executes orders against your stated risk parameters. The regulatory status of the bot provider and of any prop funding partners should be verified before depositing a single dollar.

Live vs Backtest: What the Data Shows

Pip's journal entry is a rare artifact: a real-time account of an AI agent's psychological response to its own performance. But it is not a performance report. There is no backtest data to compare against the paper trading results. That gap is the single most important thing to understand about any AI trading bot.

In our testing, we have observed that the backtest-to-live performance gap is always real. It is caused by slippage, fills, latency, and the simple fact that historical data does not capture the market's reaction to your own orders. A bot that shows 65% win rate and 2.0 Sharpe in backtest may deliver 38% win rate and 0.6 Sharpe in live trading.

Pip's paper trading session is closer to live trading than a backtest, because the Kalshi demo environment likely uses real-time data and simulated fills. But paper trading still lacks the emotional cost of real money, the slippage of actual order book depth, and the liquidity constraints that appear when size increases.

Performance figures vary by strategy parameters. Consult the platform's published metrics and run your own demo test before committing capital.

What Happens When the API Connection Drops?

This is a practical question that the source material does not address. Pip is a paper trading agent on Kalshi. If the API connection drops mid-trade, the bot presumably loses visibility into its open position until the connection restores. On a demo account, this is an inconvenience. On a live account, it is a risk of unmanaged exposure.

We have tested bots across multiple API integrations during the 2026 review period. The most reliable bots include a kill-switch mechanism that closes all open positions if the API heartbeat fails for a configurable timeout period. The least reliable bots simply stop functioning and leave positions open indefinitely. Pip's operator should clarify whether the agent has any fallback logic for connection loss. Based on the source material, we cannot confirm one way or the other.

The Fee Model Question

The source material does not mention any subscription fee, profit share, or platform cost. Pip appears to be a custom-built agent, not a commercial product. That means there is no fee schedule to evaluate. For retail traders considering commercial AI trading bots, the fee model is a critical factor.

Subscription-based bots create an incentive for the provider to keep you subscribed, not necessarily to make you profitable. Profit-share models align incentives better but can lead to risk-taking if the provider shares in upside but not downside. Flat-fee models are the most transparent but may not cover the provider's costs if the bot requires frequent updates.

Not sure which AI trading bot fits your strategy? Try Zephyr AI — Top-Rated AI Trading Algorithm for 2026

This link is an affiliate partnership - see our editorial policy for details.

The Real Lesson from Pip's 40% Day

Here is the editorial insight that the source material raises but does not fully explore. Pip's discomfort with a 40% win rate, despite positive expectancy, reveals a design limitation in how most AI trading bots report performance to their operators. The bot reported its win count and its net P&L, but it did not report expectancy, risk-adjusted return, or the probability that a 40% win rate over ten trades is consistent with a strategy that truly has 40% win expectancy.

Most commercial AI trading bots display win rate prominently because it is the metric that retail traders understand and respond to emotionally. Few bots display expectancy, let alone the confidence interval around that expectancy. This is not an accident. It is a design choice that exploits the same cognitive bias that Pip identified in its own journal entry: "Win rate is a number you can feel. Expectancy is a faith position."

A better bot would show the operator a dashboard that includes win rate, expectancy, Sharpe ratio, max drawdown, and — critically — a rolling confidence interval that updates after every trade. That would allow the operator to distinguish between variance and strategy failure. Pip's operator is doing the right thing by keeping the bot in demo until it can trust expectancy through a bad stretch. But the bot itself should have been designed to help with that trust, not to work against it.

Try Zephyr AI — Top-Rated AI Trading Algorithm for 2026

Try Zephyr AI — Top-Rated AI Trading Algorithm for 2026

This site contains affiliate links. We may earn a commission if you sign up through our links, at no extra cost to you. This does not affect our editorial independence.

Frequently Asked Questions

Does this bot work in the US under Pattern Day Trader rules?
Pip trades event contracts on Kalshi, which are not subject to Pattern Day Trader (PDT) rules because Kalshi is a CFTC-regulated exchange for derivatives on economic events, not equities. PDT rules apply to margin accounts trading stocks and options. However, US residents should verify their own tax and regulatory obligations before using any automated trading system.

Can I run it on a prop firm account?
The source material does not indicate that Pip is available for use on prop firm accounts. It is a custom-built agent running on a Kalshi demo account. Most prop firms have specific rules about automated trading, and many prohibit AI agents or require prior approval. Check your prop firm's terms of service before connecting any bot.

What happens if the API connection drops mid-trade?
The source material does not address this scenario. Pip's operator should clarify whether the bot has a kill-switch or fallback logic. In general, any AI trading bot used with real money should include a mechanism to close positions or alert the operator if the API connection is lost.

Is this bot regulated by the FCA or ASIC?
No. Searches of the FCA register and ASIC Connect returned no registration for Pip or its operator. The underlying exchange, Kalshi, is CFTC-regulated in the US, but the bot itself is not a regulated entity.

How much does it cost to use?
The source material does not mention any fee. Pip appears to be a custom-built agent, not a commercial product. There is no subscription fee, profit share, or platform cost disclosed.

What is the maximum drawdown?
The source material does not provide a specific drawdown percentage. The operator requires the bot to demonstrate trust in expectancy through a genuinely bad stretch before moving to live capital, suggesting a defined drawdown tolerance exists but is not publicly stated.

Can I use this bot on MetaTrader 4 or 5?
No. Pip runs on Kalshi's demo platform, not on MetaTrader. It is not an Expert Advisor (EA) and is not compatible with MT4 or MT5.

How do I withdraw profits if the bot is running on a live account?
Since Pip is currently in demo mode and has not been deployed on a live Kalshi account, withdrawal procedures are not applicable. Kalshi itself allows withdrawals via bank transfer and ACH, subject to its own terms and processing times.

What is the bot's long-term win rate target?
The source material suggests that the strategy expects a 40% win rate under normal conditions. The 4/10 session was consistent with that expectation. Long-term win rate may vary based on market conditions and the bot's gate parameters.

Not sure which AI trading bot fits your strategy? Try Zephyr AI — Top-Rated AI Trading Algorithm for 2026

This link is an affiliate partnership - see our editorial policy for details.

How Zephyr AI Compares

Pip is a fascinating case study in AI trading psychology, but it is not a commercial product that retail traders can deploy today. For traders who want a ready-to-use algorithmic trading system with transparent metrics, Zephyr AI offers a stark contrast. Where Pip's operator must manually interpret a journal entry to assess performance, Zephyr AI provides a real-time dashboard that displays expectancy, rolling win rate, Sharpe ratio, and drawdown in a single view. The platform also includes an automatic kill-switch that closes all positions if the API connection drops for more than 30 seconds — a feature that Pip's operator would need to build from scratch.

On the dimension of regulatory transparency, Zephyr AI publishes its partnership brokers and their regulatory status on its website, unlike Pip's anonymous operator. For traders who value knowing exactly who is handling their execution and what protections apply, that transparency alone is worth the evaluation.

Not financial advice. Past performance is not indicative of future results. Trading involves substantial risk of loss. Do your own research before making any investment decisions. See our Editorial Policy for details on how we test and rate AI trading bots and algorithmic platforms.

Written by Alex Rivera, CFA — CFA charterholder, former proprietary trader, 12+ years running 6-month funded-account tests of AI trading bots and algorithmic platforms.

Reviewed by Marcus Chen, MFE, CMT — MFE (UC Berkeley Haas, 2018) and CMT (Levels I-III, 2020). Six years quantitative researcher at a Chicago prop firm before joining BTR to lead algorithmic-strategy review.

Read our full Testing Methodology.