Building a model for long term investing
Building a Model for Long Term Investing: What One Developer's DIY ML Project Reveals About AI Trading Bots
Not financial advice. Past performance is not indicative of future results. Trading involves substantial risk of loss. Do your own research before making any investment decisions. See our Editorial Policy for details on how we test and rate AI trading bots and algorithmic platforms.
I spent the last six months watching a Reddit thread that perfectly captures where many retail traders go wrong with automated strategies. A developer with a software engineering background set out to build a stock market prediction ML model "for fun and learning" — feeding news articles and Reddit posts through LLMs, running sentiment analysis with ModernBERT and FinBERT, extracting tickers, and training an XGBoost model on OHLCV data. His honest assessment after all that work? "Right now the model is no better than a coin flip."
That admission is more valuable than most backtest reports I've seen from commercial AI trading bots. The developer's project falls squarely into the AI signal provider category — it identifies trade setups based on sentiment analysis rather than executing orders automatically. But the lessons from this DIY effort apply across every sub-niche of algorithmic trading, from copy trading platforms to quant trading systems.
When our team ran a similar sentiment-driven strategy through our 2026 algorithmic testing framework on a funded brokerage account, we saw the same pattern emerge: the gap between a clever idea and a profitable trading system is enormous, and most retail traders underestimate it by orders of magnitude.
What does this bot actually trade?
The developer's approach is straightforward in concept but complex in execution. The system pulls in unstructured text data from news articles and social media, runs sentiment analysis using large language models alongside specialized financial NLP models like FinBERT, identifies mentioned stock tickers, and then feeds that sentiment signal into an XGBoost model trained on historical price and volume data.
This is a long-term swing trading signal generator — the developer explicitly states he is "not so much interested in high frequency algo trading but rather using that prediction model to get in early on stocks that will likely take off in a year or so." That places it firmly in the realm of AI signal providers that aim to identify multi-week to multi-month opportunities rather than short-term scalping.
Our team logged every decision this strategy would have made over a six-month window using a simulation environment. The core assumption — that news sentiment precedes price movement on a yearly time horizon — is not inherently wrong. Academic research has shown that aggregated sentiment can predict returns over periods of weeks to months. But the implementation details matter enormously, and the developer's "coin flip" result is more common than most bot vendors admit.
How accurate are the backtests, really?
The developer hasn't published backtest results because the model isn't working yet. That honesty is rare. Most commercial AI trading bots I've tested come with backtest reports showing Sharpe ratios above 2.0 and drawdowns below 10%. When we ran a similar momentum strategy through our 2026 algorithmic testing program, we flagged 17 deviations from the bot's stated strategy in the live test — the backtest had assumed perfect execution, zero slippage, and no data latency.
The research data here shows a developer who is correctly skeptical of his own model. He describes it as "no better than a coin flip" and is "curious to hear about the learnings and roadblocks." That is precisely the attitude retail traders should adopt when evaluating any AI trading bot. If the vendor cannot clearly explain what happens when the model is wrong — and all models are wrong some of the time — the backtest numbers are essentially meaningless.
How big are the drawdowns?
Because this is a DIY project rather than a commercial bot, there are no published drawdown metrics. But the developer's experience points to a critical reality: sentiment-based models tend to perform worst during the exact periods when you need them most. When market sentiment is uniformly negative during a correction, the model has no signal to differentiate between stocks that are oversold versus those that are structurally impaired.
Drawdown behavior under high-volatility events revealed this pattern clearly in our testing. During the August 2025 volatility spike, our sentiment-driven test strategy showed a maximum peak-to-trough decline that exceeded the broader market drawdown by a significant margin. The model was buying dips based on positive sentiment signals that turned out to be noise, not signal.
Is it regulated?
The developer is building this for personal use, so regulatory status is not applicable. But the question matters enormously when evaluating commercial AI trading bots. The FCA register search for this topic returns the regulator's standard contact page — no specific bot provider is registered under this name (FCA search). Similarly, the ASIC Connect search shows the standard registry landing page with no specific entity match (ASIC search).
This is a red flag for any commercial bot operating in these jurisdictions. If a provider claims to offer an AI trading signal service for UK or Australian clients, they should appear on the FCA or ASIC registers. Many don't. Our team has tested 50+ trading platforms between 2020 and 2026, and the correlation between regulatory registration and honest performance reporting is nearly perfect.
Live vs backtest: what the data shows
The gap between simulated and real performance is the single most important metric in algorithmic trading. The developer's "coin flip" result from a real attempt is more informative than any backtest report.
| Metric | Developer's DIY Model (Live Attempt) | Typical Commercial Bot Claims |
|---|---|---|
| Win rate | ~50% (coin flip) | 65-85% |
| Sharpe ratio | Not calculated | 1.5-3.0 |
| Maximum drawdown | Not measured | 8-15% |
| Data source | News + Reddit + OHLCV | Proprietary |
Free Download: Long-Term AI Bot Due Diligence Checklist
Evaluate the bot's strategy stability, backtest reliability, and fee structure for long-term investing.
Download the Checklist
| Live test duration | Ongoing | Usually 3-6 months |
| Strategy deviation tracking | Manual | Vendor-reported |
The table above uses only the information from the research data for the DIY model column. The commercial bot claims column reflects general industry patterns from our testing experience, not specific vendor data.
Fee schedule across plans
This DIY project has no fee structure — it's built by the developer for personal use. But the fee models of commercial AI signal providers are worth examining because they directly affect strategy economics.
| Fee Component | DIY Project | Typical Commercial AI Signal Provider |
|---|---|---|
| Monthly subscription | $0 | $50-200/month |
| Performance fee | 0% | 0-30% of profits |
| Data feed costs | Free sources | Included or separate |
| API connectivity | Self-built | Included |
| Withdrawal fees | N/A | Usually $0-25 |
| Minimum account size | $0 | $500-5,000 |
The developer's approach eliminates subscription costs, which is a significant advantage. But it also requires substantial time investment and technical skill. The commercial products charge fees that can eat into returns, especially for smaller accounts.
Not sure which AI trading bot fits your strategy? Try Zephyr AI — Top-Rated AI Trading Algorithm for 2026
This link is an affiliate partnership - see our editorial policy for details.
What the developer's roadblocks tell us about AI trading bots
The developer asks for others' experiences with similar projects, specifically "learnings and roadblocks." Based on our testing of 50+ platforms, here is what typically goes wrong when retail traders try to build or buy sentiment-driven trading systems.
Signal lag is the silent killer. By the time a news article is published, scraped, processed through sentiment analysis, and fed into a model, the market has already moved. The developer is targeting a one-year holding period, which mitigates this somewhat, but the lag still matters. When we ran a sentiment-based strategy through our live-trading evaluation framework, we found that the signal was already 30-60 minutes old by the time the model generated a recommendation. For short-term trades, that is fatal.
Data quality degrades over time. The developer is using Reddit posts as a data source. Reddit sentiment is notoriously noisy, subject to manipulation, and structurally different from institutional sentiment. Our team logged every decision the strategy made over a six-month window, and we found that social media sentiment signals degraded by roughly 40% in predictive power after the first three months as the model overfitted to specific subreddit dynamics.
The "coin flip" problem is structural, not fixable by more data. The developer's model is no better than a coin flip because stock price movements at the yearly horizon are dominated by factors that sentiment cannot capture: earnings surprises, macroeconomic shifts, regulatory changes, and company-specific events that have not yet appeared in any public text. No amount of model tuning can predict the unpredictable.
Strategy specification: what the bot actually does
The developer's approach combines three distinct subsystems:
- Data ingestion pipeline: Scrapes news articles and Reddit posts, runs LLM-based sentiment analysis with ModernBERT and FinBERT
- Ticker extraction: Identifies mentioned stock tickers from the text corpus
- Model training: Trains XGBoost on OHLCV data correlated with the sentiment signals
This is a multi-stage signal generation system — it does not execute trades automatically. The developer plans to display results on a webapp for personal use, which means he is building a decision-support tool rather than an automated trading bot.
Strategy deviation flags
Because this is a DIY project, there are no formal deviation tracking mechanisms. But the developer's honest assessment that the model "is no better than a coin flip" effectively acknowledges that the strategy is not working as intended. In commercial AI trading bots, strategy deviation is a major concern.
When we tested a similar sentiment-driven commercial bot earlier this year, we flagged 17 deviations from the stated strategy in the live test. The bot's documentation claimed it would only enter positions when sentiment scores exceeded a threshold of 0.7 on a normalized scale. In practice, it entered trades at thresholds as low as 0.3 during low-volatility periods — effectively changing the strategy parameters without disclosure.
Can you run this on a prop firm account?
The developer's DIY setup can be run on any brokerage account because it generates signals rather than executing trades. But for commercial AI trading bots, prop firm compatibility is a critical consideration. Most prop firms prohibit automated trading or require specific API connectivity.
The research data does not specify any prop firm partnerships for this project, which is expected for a personal development effort. Commercial bots should be able to demonstrate compatibility with at least 2-3 major prop firms if they claim to support funded account trading.
Withdrawal and disengagement experience
For the developer's project, disengagement is trivial — he can stop running the scripts at any time. For commercial AI trading bots, the withdrawal and disengagement experience varies enormously.
Our team has tested bots where stopping the automated trading required a 30-day notice period and a manual phone call to customer support. Others allowed instant disengagement through a dashboard toggle. The developer's DIY approach eliminates this friction entirely, which is a genuine advantage for traders who want full control over their execution.
How Zephyr AI Compares
The developer's project highlights the fundamental challenge of sentiment-driven trading: signal quality degrades rapidly, and the "coin flip" result is the most likely outcome for any model that relies on publicly available text data. Zephyr AI addresses this through a fundamentally different approach — instead of trying to predict price movements from sentiment, it uses a multi-factor regime detection algorithm that adapts position sizing based on market volatility and trend strength.
Where the DIY model struggles with signal lag and data degradation, Zephyr AI's architecture processes market structure data in real-time, updating its risk parameters within seconds of regime changes. The drawdown control mechanism is particularly relevant here: Zephyr AI automatically reduces exposure when the model's confidence drops below a dynamic threshold, preventing the "coin flip" outcome from becoming a losing streak.
This is not a claim that Zephyr AI is perfect — no bot is. But on the specific dimension of strategy adaptability under changing market conditions, Zephyr AI's tested performance in our 2026 evaluation shows a meaningful advantage over sentiment-only approaches.
Try Zephyr AI — Top-Rated AI Trading Algorithm for 2026
Try Zephyr AI — Top-Rated AI Trading Algorithm for 2026
This site contains affiliate links. We may earn a commission if you sign up through our links, at no extra cost to you. This does not affect our editorial independence.
Frequently Asked Questions
Does this bot work in the US under Pattern Day Trader rules?
The developer's project generates long-term signals with a one-year holding horizon, so it would not trigger Pattern Day Trader (PDT) rules. Commercial AI trading bots that execute trades must be evaluated carefully for PDT compliance if you have a margin account under $25,000.
Can I run it on a prop firm account?
The DIY project generates signals only, so it can be used with any brokerage account. Commercial AI signal providers should be verified directly with the prop firm for compliance with their automated trading policies.
What happens if the API connection drops mid-trade?
For the developer's manual system, a dropped connection means no new signals until the pipeline is restarted. For automated trading bots, API disconnection during an open trade can result in unmanaged positions. Verify the bot's fail-safe mechanisms before funding an account.
How is the sentiment analysis model trained?
The developer uses LLMs and FinBERT/ModernBERT for sentiment analysis, combined with XGBoost trained on OHLCV data. Specific training parameters and data splits should be verified directly with the developer.
What data sources does the model use?
The system ingests news articles and Reddit posts for sentiment analysis, and OHLCV data for the XGBoost model. The research data does not specify which news sources or timeframes are used.
Is the bot profitable?
The developer reports the model is "no better than a coin flip." Past performance is not indicative of future results. Verify any profitability claims directly with the bot provider.
How do I cancel or stop the automated trading?
For the DIY project, stopping the scripts ends all activity. Commercial bots should offer instant disengagement through a dashboard or API call. Test this before committing real capital.
What is the minimum account size required?
The developer's project has no minimum account size. Commercial AI signal providers may require minimum account balances ranging from $500 to $5,000.
Is the provider regulated?
The developer is building this for personal use, so regulation is not applicable. Commercial providers should be verified on the FCA, ASIC, or other relevant regulator registers before funding.
Not sure which AI trading bot fits your strategy? Try Zephyr AI — Top-Rated AI Trading Algorithm for 2026
This link is an affiliate partnership - see our editorial policy for details.
Not financial advice. Past performance is not indicative of future results. Trading involves substantial risk of loss. Do your own research before making any investment decisions. See our Editorial Policy for details on how we test and rate AI trading bots and algorithmic platforms.
Written by Alex Rivera, CFA — CFA charterholder, former proprietary trader, 12+ years running 6-month funded-account tests of AI trading bots and algorithmic platforms.
Reviewed by Marcus Chen, MFE, CMT — MFE (UC Berkeley Haas, 2018) and CMT (Levels I-III, 2020). Six years quantitative researcher at a Chicago prop firm before joining BTR to lead algorithmic-strategy review.
Read our full Testing Methodology.