This page describes how we use machine learning, and more precisely an ARIMA model, in the SlapOS monitor software release to predict the disk usage of all our machines and to be warned before a machine lacks disk space.
Introduction
Running out of disk space is one of the most common and one of the most boring incidents in system administration. It is boring because it is almost always predictable: a log directory grows a bit every day, a database keeps a bit more history every week, a backup rotation was badly configured a month ago. It is common because nobody has the time to look at hundreds of graphs every morning to find which curve is slowly going down.
At Rapid.Space we operate machines of very different sizes, from EdgePOD servers in data centres to small ORS base stations deployed at customer sites. All of them run SlapOS and all of them are monitored by the monitor software release. Rather than waiting for a disk to be full, our monitoring answers a different question: how much free space will be left on this disk in ten days? The answer is computed on the machine itself, by a small model trained on the history of that very machine.
Where the data comes from
Every SlapOS node runs a collector which periodically records system metrics in a local SQLite database called collector.db. Among those metrics, the disk table stores, for each disk partition, the number of free and used bytes together with a date and a time. Another table stores how much space each SlapOS partition, which means each hosted service, is currently using.
This is a cheap and purely local form of data collection: no central time series database, no extra agent to configure, no metric shipped over the network. The history is already there, on the machine, with several months of measurements. Machine learning only has to read it.
The check_free_disk_space promise
In SlapOS, monitoring is implemented with promises. A promise is a small Python plugin which is executed periodically and which either succeeds or fails, with a human readable message. The plugin we are interested in here is check_free_disk_space, part of slapos.toolbox.
The promise does three things at each run.
First, it compares the current free space with a threshold. When the threshold is not configured, it defaults to 5% of the total disk size, which is a reasonable value both for a small SSD in an ORS and for a multi-terabyte array in a data centre. If the current free space is below the threshold, the promise fails immediately. This part is classic monitoring.
Second, if the current situation is fine, it displays the three SlapOS partitions which consumed the most disk space during the last 24 hours. This is the information a system administrator needs first when something is growing: not only that the disk is filling up, but who is filling it up.
Third, it predicts the free space for the coming days and fails if that prediction crosses the threshold. Because this computation is heavier than a simple SQL query, the promise periodicity is one hour by default, instead of the few minutes used by lighter promises.
Predicting disk usage with ARIMA
Why ARIMA
ARIMA, which stands for AutoRegressive Integrated Moving Average, is a machine learning model for time series. Like any machine learning model, it is not programmed with rules: its parameters are estimated from the observed history, and the trained model is then used to forecast future values together with a confidence interval.
ARIMA has three properties which make it a much better fit for our use case than a neural network would be. It learns from very little data, so fourteen daily measurements are already enough to start forecasting and a machine deployed two weeks ago is already covered. It is cheap: training happens on the node itself, on a CPU, in a couple of seconds, using a few kilobytes of history, with no GPU, no cloud API and no data leaving the machine, which matters a lot at the edge where a node may sit behind a mobile link in a field. And it is explainable: the output is a number of gigabytes at a given date with a 95% confidence interval, so a system administrator can immediately judge whether the model is being reasonable, which is not the case with a black box score.
The counterpart is that ARIMA models a trend, not a business logic. It is very good at telling that a disk loses three gigabytes per day and that this will not stop by itself. It cannot know that a customer will upload two terabytes next Monday.
How the prediction is computed
The promise selects one measurement per day, always taken around the same time of the day, for the partition being monitored, and requires at least 14 days of history before predicting anything. The series is loaded into a pandas dataframe indexed by day, then fitted with the ARIMA implementation of statsmodels, using the order (1, 1, 0) with a linear trend.
The order is currently hardcoded, and this deserves an explanation. Our first implementation performed a grid search over 27 combinations of the (p, d, q) parameters in order to select the best model for each machine. With recent versions of statsmodels, all 27 combinations are really evaluated, each of them taking one to two seconds, which is far beyond the 20 seconds a promise is allowed to run. We reworked the prediction to use a single order which behaves well on disk usage series, because a prediction which never finishes is worth less than a slightly suboptimal prediction which always finishes. Finding a better way to select the order per machine, without breaking the promise time budget, is still on our list.
The fitted model then forecasts the configured number of days, ten by default. If the predicted free space at the end of that horizon is below the threshold, the promise fails with an explicit message giving the disk size and the threshold. Otherwise it simply logs the prediction, for example that a given number of gigabytes will be left in ten days, which is useful information in itself. This behaviour is covered by unit tests in slapos.toolbox.
Deployment in the monitor software release
None of this has to be installed by hand. The libraries needed by the prediction, which are statsmodels, scipy and pandas, are declared as an optional prediction extra of slapos.toolbox, and the monitor software release requests them explicitly. The promise itself is instantiated by instance-node-monitoring.jinja2.cfg, where every parameter is exposed as an instance parameter: the periodicity, the threshold, the number of days predicted, whether to display the biggest partitions and whether to enable the prediction at all.
In other words, requesting one instance of the monitor software release on a node is enough to get a trained, self-updating disk usage forecast on that node, published in the monitoring interface next to all the other promises. The same software release, the same code and the same model run on an ORS in the field and on a server in a data centre.
Automation, and where it stops
A failing promise in SlapOS can trigger a reconfiguration of the instance, what we call a bang. This promise deliberately does not do that. When a disk is going to be full, the cause is almost never the monitor instance itself but another partition on the same machine, and the right answer may be to delete old backups, to move a service to another node or to provide more disk to the customer. Those are decisions, not reconfigurations. The promise therefore reports an anomaly only after three consecutive failures, and asks for a human.
This is, for us, the honest scope of automation in system management. Machine learning did not remove the system administrator, it removed the part of the work which was mechanical: watching hundreds of curves, extrapolating them by eye and discovering too late that a disk had been filling up linearly for three weeks. What reaches a human being now is a short sentence saying which machine will lack disk space, when, and which partition is responsible.
The same approach can be applied to other resources whose exhaustion is progressive rather than sudden, such as inodes, memory, bandwidth quotas or certificate lifetimes. Everything described here is open source and the prediction fits in a single readable Python file. Comments and merge requests are welcome on lab.nexedi.com.