@@ -63,6 +62,9 @@
Anomalize Methods
+
+ Reduce Forecast Error by Cleaning Anomalies
+
@@ -85,21 +87,22 @@
+
-
+
-
Anomalize Methods
Business Science
-2018-04-17
+2019-09-20
- Source:vignettes/anomalize_methods.Rmd
+ Source: vignettes/anomalize_methods.Rmd
+ anomalize_methods.RmdAnomaly detection is critical to many disciplines, but possibly none more important than in time series analysis. A time series is the sequential set of values tracked over a time duration. The definition we use for an anomaly is simple: an anomaly is something that happens that (1) was unexpected or (2) was caused by an abnormal event. Therefore, the problem we intend to solve with anomalize is providing methods to accurately detect these “anomalous” events.
The methods that anomalize uses can be separated into two main tasks:
-
@@ -129,7 +132,7 @@
STL: The STL trend follows a smoothed Loess with a Loess trend window at 91 days (as defined by
trend = "3 months"). The remainder of the decomposition is centered.
@@ -282,13 +285,13 @@ - +
Learning Lab 18 - Time Series Anomaly Detection with
anomalize
+Learning Lab 17 - Anomaly Detection with H
2OMachine Learning
+- Anomalize Methods +
- + Reduce Forecast Error by Cleaning Anomalies +
-
@@ -85,21 +87,22 @@
+
-++#> Groups: package [15] +#> $ package <chr> "tidyr", "tidyr", "tidyr", "tidyr", "tidyr", "tidy… +#> $ date <date> 2017-01-01, 2017-01-02, 2017-01-03, 2017-01-04, 2… +#> $ count <dbl> 873, 1840, 2495, 2906, 2847, 2756, 1439, 1556, 367… +#> $ observed <dbl> 8.730000e+02, 1.840000e+03, 2.495000e+03, 2.906000… +#> $ season <dbl> -2761.4637, 901.0113, 1459.7147, 1429.7532, 1238.7… +#> $ trend <dbl> 5052.582, 5046.782, 5040.983, 5035.183, 5029.383, … +#> $ remainder <dbl> -1418.11842, -4107.79368, -4005.69726, -3558.93599… +#> $ remainder_l1 <dbl> -3747.904, -3747.904, -3747.904, -3747.904, -3747.… +#> $ remainder_l2 <dbl> 3707.661, 3707.661, 3707.661, 3707.661, 3707.661, … +#> $ anomaly <chr> "No", "Yes", "Yes", "No", "No", "No", "No", "No", … +#> $ recomposed_l1 <dbl> -1456.786, 2199.890, 2752.793, 2717.032, 2520.278,… +#> $ recomposed_l2 <dbl> 5998.779, 9655.454, 10208.358, 10172.597, 9975.843…-
Anomalize Quick Start Guide
Business Science
-2018-04-17
+2019-09-20
- Source:vignettes/anomalize_quick_start_guide.Rmd+ Source:vignettes/anomalize_quick_start_guide.Rmd+anomalize_quick_start_guide.RmdThe
anomalizepackage is a feature rich package for performing anomaly detection. It’s geared towards time series analysis, which is one of the biggest needs for understanding when anomalies occur. We have a quick start section called “5-Minutes to Anomalize” for those looking to jump right in. We also have a detailed section on parameter adjustment for those looking to understand what nobs they can turn. Finally, for those really looking to get under the hood, we have another vignette called “Anomalize Methods” that gets into a deep discussion on STL, Twitter, IQR and GESD methods that are used to poweranomalize.+#> 1 2017-01-01 873 tidyr +#> 2 2017-01-02 1840 tidyr +#> 3 2017-01-03 2495 tidyr +#> 4 2017-01-04 2906 tidyr +#> 5 2017-01-05 2847 tidyr +#> 6 2017-01-06 2756 tidyr +#> 7 2017-01-07 1439 tidyr +#> 8 2017-01-08 1556 tidyr +#> 9 2017-01-09 3678 tidyr +#> 10 2017-01-10 7086 tidyr +#> # … with 6,365 more rows@@ -112,9 +115,9 @@
5-Minutes To Anomalize
Load libraries.
- +Get some data. We’ll use the
tidyverse_cran_downloadsdata set that comes withanomalize. A few points:It’s a
tibbletimeobject (classtbl_time), which is the object structure thatanomalizeworks with because it’s time aware! Tibbles (classtbl_df) will automatically be converted.
@@ -127,17 +130,17 @@
#> # Groups: package [15] #> date count package #> <date> <dbl> <chr> -#> 1 2017-01-01 873. tidyr -#> 2 2017-01-02 1840. tidyr -#> 3 2017-01-03 2495. tidyr -#> 4 2017-01-04 2906. tidyr -#> 5 2017-01-05 2847. tidyr -#> 6 2017-01-06 2756. tidyr -#> 7 2017-01-07 1439. tidyr -#> 8 2017-01-08 1556. tidyr -#> 9 2017-01-09 3678. tidyr -#> 10 2017-01-10 7086. tidyr -#> # ... with 6,365 more rows
We can use the general workflow for anomaly detection, which involves three main functions:
-
@@ -155,18 +158,19 @@
tidyverse_cran_downloads_anomalized %>% glimpse() #> Observations: 6,375 #> Variables: 12 -#> $ package <chr> "tidyr", "tidyr", "tidyr", "tidyr", "tidyr", "ti... -#> $ date <date> 2017-01-01, 2017-01-02, 2017-01-03, 2017-01-04,... -#> $ count <dbl> 873, 1840, 2495, 2906, 2847, 2756, 1439, 1556, 3... -#> $ observed <dbl> 8.730000e+02, 1.840000e+03, 2.495000e+03, 2.9060... -#> $ season <dbl> -2761.4637, 901.0113, 1459.7147, 1429.7532, 1238... -#> $ trend <dbl> 5052.582, 5046.782, 5040.983, 5035.183, 5029.383... -#> $ remainder <dbl> -1418.11842, -4107.79368, -4005.69726, -3558.935... -#> $ remainder_l1 <dbl> -3747.904, -3747.904, -3747.904, -3747.904, -374... -#> $ remainder_l2 <dbl> 3707.661, 3707.661, 3707.661, 3707.661, 3707.661... -#> $ anomaly <chr> "No", "Yes", "Yes", "No", "No", "No", "No", "No"... -#> $ recomposed_l1 <dbl> -1456.786, 2199.890, 2752.793, 2717.032, 2520.27... -#> $ recomposed_l2 <dbl> 5998.779, 9655.454, 10208.358, 10172.597, 9975.8...
Let’s explain what happened:
-
@@ -210,7 +214,7 @@
Adjusting the trend and seasonality are fundamental to time series analysis and specifically time series decomposition. With
anomalize, it’s simple to make adjustments because everything is done with date or datetime information so you can intuitively select increments by time spans that make sense (e.g. “5 minutes” or “1 month”).To get started, let’s isolate one of the time series packages: lubridate.
+#> 1 2017-01-01 643 lubridate +#> 2 2017-01-02 1350 lubridate +#> 3 2017-01-03 2940 lubridate +#> 4 2017-01-04 4269 lubridate +#> 5 2017-01-05 3724 lubridate +#> 6 2017-01-06 2326 lubridate +#> 7 2017-01-07 1107 lubridate +#> 8 2017-01-08 1058 lubridate +#> 9 2017-01-09 2494 lubridate +#> 10 2017-01-10 3237 lubridate +#> # … with 415 more rowslubridate_daily_downloads <- tidyverse_cran_downloads %>% - filter(package == "lubridate") %>% + filter(package == "lubridate") %>% ungroup() lubridate_daily_downloads @@ -218,17 +222,17 @@#> # Index: date #> date count package #> <date> <dbl> <chr> -#> 1 2017-01-01 643. lubridate -#> 2 2017-01-02 1350. lubridate -#> 3 2017-01-03 2940. lubridate -#> 4 2017-01-04 4269. lubridate -#> 5 2017-01-05 3724. lubridate -#> 6 2017-01-06 2326. lubridate -#> 7 2017-01-07 1107. lubridate -#> 8 2017-01-08 1058. lubridate -#> 9 2017-01-09 2494. lubridate -#> 10 2017-01-10 3237. lubridate -#> # ... with 415 more rows
Next, let’s perform anomaly detection.
+#> $ date <date> 2017-01-01, 2017-01-02, 2017-01-03, 2017-01-04, 2… +#> $ observed <dbl> 6.430000e+02, 1.350000e+03, 2.940000e+03, 4.269000… +#> $ season <dbl> -2077.6548, 517.9370, 1117.0490, 1219.5377, 865.11… +#> $ trend <dbl> 2474.491, 2491.126, 2507.761, 2524.397, 2541.032, … +#> $ remainder <dbl> 246.1636, -1659.0632, -684.8105, 525.0657, 317.851… +#> $ remainder_l1 <dbl> -3323.425, -3323.425, -3323.425, -3323.425, -3323.… +#> $ remainder_l2 <dbl> 3310.268, 3310.268, 3310.268, 3310.268, 3310.268, … +#> $ anomaly <chr> "No", "No", "No", "No", "No", "No", "No", "No", "N… +#> $ recomposed_l1 <dbl> -2926.58907, -314.36218, 301.38509, 420.50889, 82.… +#> $ recomposed_l2 <dbl> 3707.105, 6319.331, 6935.079, 7054.202, 6716.417, …lubridate_daily_downloads_anomalized <- lubridate_daily_downloads %>% time_decompose(count) %>% @@ -240,16 +244,16 @@lubridate_daily_downloads_anomalized %>% glimpse() #> Observations: 425 #> Variables: 10 -#> $ date <date> 2017-01-01, 2017-01-02, 2017-01-03, 2017-01-04,... -#> $ observed <dbl> 6.430000e+02, 1.350000e+03, 2.940000e+03, 4.2690... -#> $ season <dbl> -2077.6548, 517.9370, 1117.0490, 1219.5377, 865.... -#> $ trend <dbl> 2474.491, 2491.126, 2507.761, 2524.397, 2541.032... -#> $ remainder <dbl> 246.1636, -1659.0632, -684.8105, 525.0657, 317.8... -#> $ remainder_l1 <dbl> -3323.425, -3323.425, -3323.425, -3323.425, -332... -#> $ remainder_l2 <dbl> 3310.268, 3310.268, 3310.268, 3310.268, 3310.268... -#> $ anomaly <chr> "No", "No", "No", "No", "No", "No", "No", "No", ... -#> $ recomposed_l1 <dbl> -2926.58907, -314.36218, 301.38509, 420.50889, 8... -#> $ recomposed_l2 <dbl> 3707.105, 6319.331, 6935.079, 7054.202, 6716.417...
Learning Lab 18 - Time Series Anomaly Detection with
anomalize
+Learning Lab 17 - Anomaly Detection with H
2OMachine Learning
+- Split the
datainto training and testing data that maintains the correct time-series sequence using thepropargument.
+ - Models the daily time series of the training data set from observed (demonstrates no cleaning) or observed and cleaned (demonstrates improvement from cleaning). Specified by the
col_trainargument.
+ - Compares the predictions to the observed values. Specified by the
col_testargument.
+ Learning Lab 18 - Time Series Anomaly Detection with
anomalize
+Learning Lab 17 - Anomaly Detection with H
2OMachine Learning
+- Anomalize Methods +
- + Reduce Forecast Error by Cleaning Anomalies +
-
@@ -86,14 +98,9 @@
- -- - - -- ++
anomalize
@@ -112,9 +119,9 @@ Installation
-You can install the development version with
+devtoolsor the most recent CRAN version withinstall.packages():You can install the development version with
+install.packages("anomalize")devtoolsor the most recent CRAN version withinstall.packages():@@ -133,8 +140,8 @@
Getting Started
Load the
- +tidyverseandanomalizepackages.Next, let’s get some data.
anomalizeships with a data set calledtidyverse_cran_downloadsthat contains the daily CRAN download counts for 15 “tidy” packages from 2017-01-01 to 2018-03-01.@@ -388,5 +393,6 @@tidyverse_cran_downloads %>% ggplot(aes(date, count)) + @@ -158,7 +165,7 @@If you’re familiar with Twitter’s
AnomalyDetectionpackage, you can implement that method by combiningtime_decompose(method = "twitter")withanomalize(method = "gesd"). Additionally, we’ll adjust thetrend = "2 months"to adjust the median spans, which is how Twitter’s decomposition method works.@@ -380,7 +385,7 @@# Get only lubridate downloads lubridate_dloads <- tidyverse_cran_downloads %>% - filter(package == "lubridate") %>% + filter(package == "lubridate") %>% ungroup() # Anomalize!! @@ -190,7 +197,7 @@time_frequency()andtime_trend()for generating frequency and trend spans using date and datetime information, which is more intuitive than selecting numeric values. Also,period = "auto"automatically selects frequency and trend spans based on the time scale of the data.# Time Frequency + @@ -202,8 +209,8 @@plot_anomaly_decomposition()for visualizing the inner workings of how algorithm detects anomalies in the “remainder”.- +tidyverse_cran_downloads %>% - filter(package == "lubridate") %>% +-tidyverse_cran_downloads %>% + filter(package == "lubridate") %>% ungroup() %>% time_decompose(count) %>% anomalize(remainder) %>% @@ -214,10 +221,10 @@- Vector functions for anomaly detection:
iqr()andgesd(). These are great for just using on numeric data. Note that trend and seasonality should already be removed for time series data.
@@ -370,8 +376,7 @@# Data with outliers -set.seed(100) -x <- rnorm(100) -idx_outliers <- sample(100, size = 5) +# Data with outliers +set.seed(100) +x <- rnorm(100) +idx_outliers <- sample(100, size = 5) x[idx_outliers] <- x[idx_outliers] + 10 # IQR method @@ -235,7 +242,7 @@- Anomaly Reports: Using
verbose = TRUE, we can return a nice report of useful information related to the outliers.
lubridate_dloads %>% +-lubridate_dloads %>% time_decompose(count) %>% anomalize(remainder, verbose = TRUE) #> $anomalized_tbl @@ -337,10 +344,9 @@- Twitter’s
AnomalyDetection, which implements decomposition using median spans and the Generalized Extreme Studentized Deviation (GESD) test for anomalies. -
-
forecast::tsoutliers()function, which implements the IQR method.
+
forecast::tsoutliers()function, which implements the IQR method.Developers
Developers
Developers
+ diff --git a/docs/news/index.html b/docs/news/index.html index dabd5c5..12ab497 100644 --- a/docs/news/index.html +++ b/docs/news/index.html @@ -1,6 +1,6 @@ - + @@ -8,28 +8,41 @@Changelog • anomalize + - + - + + - + - + + - + + + + + - + + + + + + - + + - + + - @@ -59,14 +69,15 @@- -+ -++-Changelog
Source:NEWS.md++-+anomalize 0.2.0 Unreleased +
+-
+
-
+
clean_anomalies()- A new function to simplify cleaning anomalies by replacing with trend and seasonal components. This is useful in preparing data for forecasting.
+
1.A. STL
-The STL method uses the stl() function from the stats package. STL works very well in circumstances where a long term trend is present. The Loess algorithm typically does a very good job at detecting the trend. However, it circumstances when the seasonal component is more dominant than the trend, Twitter tends to perform better.
+The STL method uses the stl() function from the stats package. STL works very well in circumstances where a long term trend is present. The Loess algorithm typically does a very good job at detecting the trend. However, it circumstances when the seasonal component is more dominant than the trend, Twitter tends to perform better.
@@ -140,18 +143,18 @@
1.C. Comparison of STL and Twitter Decomposition Methods
Load two libraries to perform the comparison.
-
+
Collect data on the daily downloads of the lubridate package. This comes from the data set, tidyverse_cran_downloads that is part of anomalize package.
# Data on `lubridate` package daily downloads
lubridate_download_history <- tidyverse_cran_downloads %>%
- filter(package == "lubridate") %>%
+ filter(package == "lubridate") %>%
ungroup()
# Output first 10 observations
lubridate_download_history %>%
- head(10) %>%
- knitr::kable()
+ head(10) %>%
+ knitr::kable()
1.A. STL
-The STL method uses the stl() function from the stats package. STL works very well in circumstances where a long term trend is present. The Loess algorithm typically does a very good job at detecting the trend. However, it circumstances when the seasonal component is more dominant than the trend, Twitter tends to perform better.
The STL method uses the stl() function from the stats package. STL works very well in circumstances where a long term trend is present. The Loess algorithm typically does a very good job at detecting the trend. However, it circumstances when the seasonal component is more dominant than the trend, Twitter tends to perform better.
@@ -140,18 +143,18 @@
1.C. Comparison of STL and Twitter Decomposition Methods
1.C. Comparison of STL and Twitter Decomposition Methods
Load two libraries to perform the comparison.
- +Collect data on the daily downloads of the lubridate package. This comes from the data set, tidyverse_cran_downloads that is part of anomalize package.
# Data on `lubridate` package daily downloads
lubridate_download_history <- tidyverse_cran_downloads %>%
- filter(package == "lubridate") %>%
+ filter(package == "lubridate") %>%
ungroup()
# Output first 10 observations
lubridate_download_history %>%
- head(10) %>%
- knitr::kable()| date | @@ -239,7 +242,7 @@
|---|
