splunk
Fastly (Signal Sciences) NG-WAF - TA
This app is NOT supported by Splunk. Please read about what that means for you here.
Overview
Details
For users of Fastly NG-WAF who would like to enrich their Splunk data with information from the Fastly NG-WAF. This app enables the importing of Events, Activity, and raw request information to Splunk.
This is an open source project, no support provided, public repository is available and installation documentation can be found at https://github.com/fastly/sigsci-splunk-app. The best way to report issues with the app is to report an issue (https://github.com/fastly/sigsci-splunk-app/issues) on the github page so that it can be tracked.
This is an open source project, no support provided, public repository is available and installation documentation can be found at https://github.com/fastly/sigsci-splunk-app. The best way to report issues with the app is to report an issue (https://github.com/fastly/sigsci-splunk-app/issues) on the github page so that it can be tracked.
Release Notes
Version 1.0.38
April 1, 2024
- Adds toggle to disabling catching up making from/until times always calculated from
now - delta - Adds configurable API connect/read timeouts to each input for the HTTP client
- Adds some extra configuration around catchup if it is enabled, either to reset to
now - delta(default) if the timestamp is older than 24 hours, or exactly 24 hours ago to comply with API restrictions - Handles the POST parameter changes to the Feed Endpoint API where pagination is now done via POST parameters rather than query params
Version 1.0.37
Nov. 6, 2023
- Refactors the time functions to now rely on
time.time()instead of thedatetimelibrary - Fixes timestamp calculations crossing timezones (See: issue #37 and potentially any other edge cases as a result of
datetimelibrary timezone calculation issues) - Reduced logging when iterating over multiple events
Version 1.0.36
Aug. 15, 2023
- Makes datetime objects timezone aware (e.g datetime.utcnow => datetime.now(timezone.utc))
- Fixes checkpoint save in requests input if there are no results returned
- Fixes a bug where if the last saved until_time could go over 24 hours ago leading to an API error
- Fixed a bug where the input module could lag permanently behind in the event the last until time was hours old
- Fixes duplicate log_info statements reducing logging
Version 1.0.35
July 10, 2023
NOTE: The prior 1.0.34 release contained an incorrect artifact. The 1.0.35 release is a re-release of 1.0.34 with the correct artifact. Please use 1.0.35 instead of 1.0.34.
- Fixed github issue #32 - checks for headersOut and headersIn and does not include them in the event output if the key is not in the response dictionary
- Improved formatting of event data
- Fixed bug where if events and requests inputs were running at the same time, one would overwrite the state of the last time
- Improved setting and handling of timestamps
- Updated AOB Vendored Libraries to that of the latest AddOnBuilder package used to create the release
- Removed deprecated variables from various inputs that are declared already in helper config
Version 1.0.34
July 6, 2023
NOTE: This 1.0.34 release contained an incorrect artifact. The 1.0.35 release is a re-release of 1.0.34 with the correct artifact. Please use 1.0.35 instead of 1.0.34.
Version 1.0.33
May 23, 2023
- Fixed the headers being duplicated into headersIn Issue 26
- Moved shared code for pulling Events from the API to the
sigsci_helper.pyto reduce introducing errors into re-used code - Moved shared code for creating the from and until times to
sigsci_helper.pyto reduce the likelyhood of introducing errors in the shared code - Added a config object to help with the moving of the re-used code to
sigsci_helper.py - Added support for using the
helper.get_check_point()andhelper.set_check_point. This way going forward the code can see what the lastuntil_timewas to make sure that it does not run overlapping time periods
Version 1.0.32
May 11, 2023
- Fixed issue with the interval being in minutes instead of seconds in the code
Version 1.0.30
April 25, 2023
- Consolidated the HTTP Requests processing to a shared module for the different inputs
- Improved the error handling for the Requests to catch HTTP errors that were being missed
Version 1.0.28
March 14, 2023
- Fix for logging out incorrect variable
Version 1.0.25
Feb. 28, 2020
Changes in the new version:
- Upgrade to the latest SDK from the Splunk Add On Builder
- Updated everything to work with Python3 in prepation for the migration to Python3 in Splunk 8.x
- Fixed the timestamp finding issue so that items from Signal Sciences will reflect the right event timestamps in Splunk
- Fixed the Sites Events endpoint to be activity instead of events so that SigSci events and Audit information for sites appears
Version 1.0.23
Aug. 12, 2019
- Updated the Site events to pull event types (audit & flags)
- Added a new Input Type of SigSci Activity, only one of these are needed to pull the Corp Events.
Version 1.0.22
July 28, 2019
Updated the endpoint for Events to use the Activity endpoint. This way both Flagged IP Alerts and Agent Alerts will be imported.
1.0.21
Fixed an issue where the Splunk App was not correctly using the multi instance mode. The symptoms would be that everything appears to run correctly but would exit before writing out the events. There is a new field in the Inputs configuration for Interval that does need to be filled in if it is not already.
Version 1.0.21
July 27, 2019
Fixed an issue where the Splunk App was not correctly using the multi instance mode. The symptoms would be that everything appears to run correctly but would exit before writing out the events. There is a new field in the Inputs configuration for Interval that does need to be filled in if it is not already.
Version 1.0.19
Dec. 13, 2018
- Added back in the props.conf as it was accidentally excluded from 1.0.18
Version 1.0.18
Dec. 9, 2018
- Added support for Proxy configuration
- Updated splunklib to the latest version
- Used new method for App configuration instead of the old setup view
- Added Help messages for the configuration options
Version 1.0.17
Oct. 5, 2018
1.0.17
- Fixed requirement for app.conf for Splunk Cloud Support, wasn't correctly fixed in 1.0.16.
1.0.15
- Updated props.conf for time issue https://github.com/dacoburn/sigsci-splunk-app/issues/2
1.0.14
- Fixed issue with timestamp not being found by adding TIME_PREFIX = timestamp . Previously the JSON could be to large for the default look ahead to find the timestamp element. This way the look ahead starts from the timestamp object.
- Linted and fixed formatting of python scripts
- Removed default Data Inputs for Requests and Events as this was keeping you from being able to delete them and caused isues.
IMPORTANT NOTE: If you had modified the default example Data Input you will need to go back and re-add the "5" for the Delta. Otherwise this will default to 0 and no data will be pulled.
New Features:
- Added support for the new API Tokens. You can either use the Username/API Token or the Username/Password combo. If both are filled in the API Token will take precedence.
Version 1.0.16
Oct. 5, 2018
1.0.16
- Fixed requirement for app.conf for Splunk Cloud Support.
1.0.15
- Updated props.conf for time issue https://github.com/dacoburn/sigsci-splunk-app/issues/2
1.0.14
- Fixed issue with timestamp not being found by adding TIME_PREFIX = timestamp . Previously the JSON could be to large for the default look ahead to find the timestamp element. This way the look ahead starts from the timestamp object.
- Linted and fixed formatting of python scripts
- Removed default Data Inputs for Requests and Events as this was keeping you from being able to delete them and caused isues.
IMPORTANT NOTE: If you had modified the default example Data Input you will need to go back and re-add the "5" for the Delta. Otherwise this will default to 0 and no data will be pulled.
New Features:
- Added support for the new API Tokens. You can either use the Username/API Token or the Username/Password combo. If both are filled in the API Token will take precedence.
Version 1.0.15
Oct. 2, 2018
1.0.15
- Updated props.conf for time issue https://github.com/dacoburn/sigsci-splunk-app/issues/2
1.0.14
- Fixed issue where the build script was not correctly updating some of the python files for the version
- Fixed issue with timestamp not being found by adding TIME_PREFIX = timestamp . Previously the JSON could be to large for the default look ahead to find the timestamp element. This way the look ahead starts from the timestamp object.
- Linted and fixed formatting of python scripts
- Removed default Data Inputs for Requests and Events as this was keeping you from being able to delete them and caused isues.
IMPORTANT NOTE: If you had modified the default example Data Input you will need to go back and re-add the "5" for the Delta. Otherwise this will default to 0 and no data will be pulled.
New Features:
- Added support for the new API Tokens. You can either use the Username/API Token or the Username/Password combo. If both are filled in the API Token will take precedence.
Version 1.0.14
Oct. 2, 2018
Bugs Fixed:
- Fixed issue where the build script was not correctly updating some of the python files for the version
- Fixed issue with timestamp not being found by adding TIME_PREFIX = timestamp . Previously the JSON could be to large for the default look ahead to find the timestamp element. This way the look ahead starts from the timestamp object.
- Linted and fixed formatting of python scripts
- Removed default Data Inputs for Requests and Events as this was keeping you from being able to delete them and caused isues.
IMPORTANT NOTE: If you had modified the default example Data Input you will need to go back and re-add the "5" for the Delta. Otherwise this will default to 0 and no data will be pulled.
New Features:
- Added support for the new API Tokens. You can either use the Username/API Token or the Username/Password combo. If both are filled in the API Token will take precedence.
Version 1.0.13
Sept. 13, 2018
This release includes improvements for:
- Changed default behavior of modular scripts from single_instance = True to single_instance = false. This means that there will be a unique execution of the script for each data input configured. This is important so that if one of the Data inputs causes an error it won't effect the other ones. Also for sites with high RPS it can potentially take to long to have the data be pulled sequentially instead of concurrently.
1.0.12 Improvements:
- Retry behavior if rate limiting for pulling details is hit
- More efficient methods for writing events. Previously as the events were detected they were being written out. Now to improve the Script performance the event write call is done after all items are pulled from SigSci
- Better error handling if the URL is not correctly built do to wrong entries being configured in the app settings
- User-Agent string update to be recognizable as the SigSci Splunk app
Version 1.0.11
Dec. 11, 2017
- Fixed issue with regression for empty response headers
- Fixed issues where upgrade backups were left in the Splunk App Folder
Version 1.0.10
Nov. 13, 2017
- Accidentally reverted the fix for requests with no Response headers from 1.08 when releasing 1.09. This fix has been added back in.
Version 1.0.9
Nov. 8, 2017
- Refactored Module Input script to support when multiple sites are setup for Requests or Events. Before it would cause things to error out.
Version 1.0.8
Oct. 5, 2017
- Removed logging out of Token when debug logging is enabled.
Version 1.0.7
Aug. 3, 2017
- Fixed issue with time calculations not being correct and leading to potential errors from the API or getting unexpected time periods pulled back for the Requests API.
- Fixed issue where output format from SigSci is not optimal for header data. Header Data will now be properly sortable. The result from the API currently returns a JSON object of a list of lists for the headers so the header entries were showing in a format of [Header, Value] instead of {"header": "value"}