Q&A: The Methodology behind AgFlow’s Forward Curves
Reading time: 5 minutes
Risk managers and traders use forward curves for various reasons:
- Risk managers to check and challenge Traders’ positions through a series of processes:
- checking the exposure
- P&L (Profit & Loss) reporting utilizing an MTM (Mark-to-Market) approach
- Determine the Value at Risk (VaR)
- Ensuring positions are correctly hedged
- And more
- For Traders, forward curves are great to assess market movement and predict (or react) to market signals
The fact, and the matter at hand, is that most forward curves are incomplete due to the lack of data availability from brokers or a low update frequency of a given commodity. This is why AgFlow developed has decided to fill the gaps of missing forward prices by reconstructing complete forward curves with forward prices computed by our Machine Learning algorithm.
- How do we collect the data ?
- What are the AgFlow forward curves?
- Why do we use the IQR method?
- Why not use other methods, as Linear Interpolation, Akima interpolation, or ARIMA?
- How do we account for seasonality or structural market changes?
- How do we backtest our data?
- Which should you use? Regular forward curves or AgFlow forward curves?
- Which tickers do AgFlow Forward Curves cover?
Get Access to 50 Forward Curves Daily
7-Day Free Trial
How do we collect the data?
The data we use to reconstruct forward curves follows a stream. We obtain cash quotes from trusted brokers or public sources, like Agropa Trading or USDA, in a variety of formats, from pdf reports to excel spreadsheets. This data then goes through the parsing process, which automatically processes data into the database in the desired format. Once stored in our database, it is ready to be used for the forward curves processing.
What are the AgFlow forward curves?
Our forward curves differ from what is found elsewhere on the market. Regular forward curves—based on forward contracts— depend on the trade volume of their commodity.
Typically, PRAs have improved these by relying on market analysts’ professional judgment, allowing them to fill some data gaps.
AgFlow goes even further. Our forward curves are an aggregation of cash and futures quotes that we complete with synthetic data.
- First, the data is preprocessed using a simple IQR method to filter outliers
- Then, quotes data are aggregated using time interpolation, and Spearman correlated futures to have the most negligible error (or incertitude) possible
- Once aggregated, the data passes through a Machine Learning algorithm to:
- Understand the dataset
- Create artificial data points predicted from training
- Test the datasets
- Back-test the curves every day to refit the data and improve the model
- It then produces the best forward curves potential, which are perfected even further with the backtesting
Original Incomplete Data for Argentina Feed Barley FOB Argentina Forward Curves
AgFlow goes even further. Our forward curves are an aggregation of cash and futures quotes that we complete with synthetic data.
Fully Reconstructed Data for Argentina Feed Barley FOB Argentina Forward Curves
Why do we use the IQR method?
Commodities data are subject to speculation and SnD like any trade in the world, and, as such, there isn’t a unique pricing for a given commodity. This leads to a wide range of prices quoted every day for the most popular commodities. Moreover, price imputation, or currency and unit conversion mistakes, can sometimes lead to extremely low or high prices.
This variety of prices also leads to some incertitude when producing forward curves. It is necessary to filter the data in the pre-processing to remove these outliers and have the least incertitude possible to ensure that the curve is not biased by extreme prices that could lead risk managers or traders to draw wrong conclusions.
However, at the same time, more complex statistical techniques can remove even more data. Those produce the opposite effect of smoothing the curves too much, which results in information loss and renders market movements and signals difficult or impossible to view.
Additionally, this pre-processing takes time to produce. But forward curves need to be reprocessed daily, and the more refined the techniques, the more times this task takes. Thus, it was necessary to find a good balance between time consumption and incertitude in curves. Testing both parameters led us to choose the IQR (Inter-Quartile-Range) method, as it was the best compromise.
Get Access to 50 Forward Curves Daily
7-Day Free Trial
Why do we NOT use the Linear Interpolation, Akima interpolation, or ARIMA method?
Before inputting the data into the Machine Learning model, we first aggregated cash and futures quotes. The reason behind this is because datasets are not complete. If incomplete, this can lead the model to be less performant and thus less accurate.
Now when aggregating the data, the dataset must follow the price curves in time. In commodities data, prices can change drastically due to seasonality or market structure changes. With this in mind, it is necessary to limit the Mean Absolute Error (MAE) value, which would artificially increase the incertitude and accuracy of prices over time.
Therefore, similarly to IQR, we tested the mean, Linear Interpolation, Akima interpolation, the ARIMA method, time interpolation, correlated futures, and a combination of the last two for MAE. The results of these tests overwhelmingly showed that a combination of time interpolation and correlated futures had the least MAE of all.
The time interpolation was also essential to account for seasonality, as fitting the data over time across several years allowed it to better fit the data’s structure.
How do we account for seasonality or structural market changes?
Seasonality is an essential mechanism in grains commodities trading and translates directly into the data. We use time interpolation to aggregate the data that will be inputted into the Machine Learning model. The model is then trained and tested on these datasets and can understand the regular pricing changes based on time.
However, structural market changes are impossible to predict by the model as these changes are produced by external factors, such as policy changes, infrastructure changes, or even trade wars and climate change. Nonetheless, the model’s flexibility also presents a remarkable opportunity since it is not based on fixed calendars nor considers the set of rules that governs grain markets. As the forward curves are back-tested every day based on the last year of data, as well as the entire historical set every three months, it adapts to the most recent changes whilst conserving knowledge of seasonality and previous structural changes, if any.
How do we backtest our data?
The curve backtesting is done on a daily basis. It consists of comparing the predicted values with the actual data and grading how well they matched. The Machine Learning model then refits the data with the recent entries and adapts its calculations according to the grading, ensuring a continuous improvement of the model.
Get Access to 50 Forward Curves Daily
7-Day Free Trial
Which should you use? Regular forward curves or AgFlow forward curves?
Both have perks, limitations, and specific use cases.
AgFlow’s forward curves are complete and give the full picture over a period of 12 months. As such, they are safer to benchmark the MTM process and to calculate VaR and P&L to challenge traders on their positions. Moreover, with forward curves, traders have a more precise tool to detect market signals and predict market movements.
On the other hand, standard forward curves are based on actual prices – whereas AgFlow’s forward curves contain large amounts of artificial prices. Consequently, a trader cannot take positions using our solution. In this case, it acts as a benchmark and is limited by its nature.
Comparison Between Data, Forward curves, and AgFlow’s Forward Curves
Which tickers are available?
There are currently 50 tickers available at launch on the platform, based on the most performant ones.
- Argentina Corn FOB Argentina
- Argentina Feed Barley FOB Argentina
- Argentina Milling Wheat 11.5 FOB Argentina
- Argentina Soymeal CIF Netherlands
- Argentina Soymeal FOB Argentina
- Argentina Soyoil CIF India
- Argentina Soyoil FOB Argentina
- Argentina Sunoil FOB Argentina
- Australia Feed Barley FOB Australia
- Australia Rapeseed FOB Australia
- Baltic Malting Barley FOB Baltic
- Brazil Corn FOB Brazil
- Brazil Soybean CFR China
- Brazil Soybean FOB Brazil
- Brazil Soymeal FOB Brazil
- Brazil Soymeal Pellet FOB Brazil
- Brazil Soyoil FOB Brazil
- Canada Rapeseed 1CDA EXW Canada
- Canada Wheat 1CWAD EXW Canada
- Canada Yellow Peas EXW Canada
- France Corn FOB France
- France Feed Barley FOB France
- France Milling Wheat 11 FOB France
- Germany Rapeseed Meal FOB Germany
- Germany Soybean Oil FOB Germany
- Indonesia Palm Oil FOB Indonesia
- Malaysia Fatty Palm Oil FOB Malaysia
- Malaysia Palm Oil FOB Malaysia
- Malaysia Palm Oil Kernels FOB Malaysia
- Malaysia Palm Olein FOB Malaysia
- Optional Corn CIF Netherlands
- Optional Feed Barley CIF Netherlands
- Optional Palm Oil CFR India
- Optional Palm Olein CFR India
- Optional Soymeal FOB Netherlands
- Optional Soyoil CFR India
- Russia Milling Wheat 12.5 CIF Turkey
- Russia Milling Wheat 12.5 FOB Russia
- Russia Wheat CPT 12.5 Russia
- Spain Wheat Feed EXW
- Ukraine Corn FOB Ukraine
- Ukraine Milling Wheat 11.5 FOB Ukraine
- Ukraine Sunoil CIF India
- Ukraine Sunoil FOB Ukraine
- US Corn FOB US Gulf
- US DDGS CFR China
- US DNS Wheat FOB US Gulf
- US DNS Wheat FOB US PNW
- US Soymeal CFR Bangladesh
- US Soymeal CFR Malaysia
