These steps needs to be done in order.
1. Get the data in. Use one of these sourcetype names "access_common", "access_combined", "iis", "ms:iis:auto", "ms:iis:default" "apache:access", "oracle:weblogic" or "aws:cloudfront:accesslogs".
2. Modify the eventtype called web-traffic so it points to the correct indexes for the web logs.
3. Configure the sites that you want to monitor.
4. Run "Generate Session" and "Generate pages" lookup searches.
5. Enable Data Model Acceleration for Web datamodel.
6. Configure goals (Optional)
The Splunk App for Web Analytics currently supports data from Apache, IIS, Oracle Weblogic and AWS Cloudfront logs. Make sure you use the sourcetypes access_common, access_combined, iis, apache:access oracle:weblogic or aws:cloudfront:accesslogs for this data. If you already have data in Splunk under a different sourcetype you can use sourcetype renaming or by modifying the eventtype web-traffic to include the names of your sourcetypes.
If you plan on using the sourcetype apache:access, you need to install the prerequisite Add-on as this app builds on top of the base field extractions from the Add-on for Apache:
https://splunkbase.splunk.com/app/3186/
If you plan on using the sourcetypes ms:iis:auto or ms:iis:default, you need to install the prerequisite Add-on as this app builds on top of the base field extractions from the Splunk Add-on for IIS:
https://splunkbase.splunk.com/app/3185/
If you plan on using the sourcetype aws:cloudfront:accesslogs, you need to install the prerequisite Add-on as this app builds on top of the base field extractions from the Splunk Add-on for Amazon Web Services:
https://splunkbase.splunk.com/app/1876/
The app comes with three sets of sample data for Apache, IIS and IIS in the w3c format, ms:iis:default. You can enable these static sample inputs by going into Settings->Data inputs->Files & Directories
Modify the eventtype called web-traffic so it points to the correct indexes for the web logs.
The Splunk App for Web Analytics works in a multi website environment. Websites are configured from a combination of the host and the source field. Each event with that unique combination will be tagged with the corresponding website name in the field "site". You can use wildcards (*) in the Source and Host field to select multiple files matching a pattern. There is a website setup form page that allows you to add these in an easy way.
Here are some examples of valid website configurations with or without wildcards
No wildcards
Site Host Source
roadrunner.com server1 /var/log/httpd/access_log
roadrunner.com server2 /var/log/httpd/access_log
With wildcards
Site Host Source
roadrunner.com server /var/log/httpd/access_
The data in the setup form will be stored in the lookup called WA_settings. You can also manually edit this lookup. The websites setup page can be found under Setup->Websites.
Once the data has been imported run the two lookups "Generate user sessions" and "Generate pages". These will be used throughout the app. Once run the first time, they will automatically be updated via two scheduled searches that runs every 10 minutes that adds any new data coming into the app. Running these lookup searches might take a long time depending on how much data you have in Splunk but its important you let the searches finish before you move on to the next step. If you have too much data to run these for everything you can modify the time period to something less than "All time" which is the default time period. The lookup reports can be found under Setup->Lookups or by using the links above. It's important that thes searches return results. If not, the app will not work.
The Splunk App for Web Analytics uses data model acceleration extensively to power the dashboards. The app allows you to select the datamodel name you want to use. By default the app uses the datamodel "Web". I you have a naming conflict for this datamodel (there is a Web data model alread in the CIM app which is slightly different to this one), you can choose to rename it. See additional instructions below for using a custom datamodel name.
Once the lookups in the previous step has completed you should enable acceleration for the data model "Web" (or the custom name you have chosen). The data model can be found under Settings->Data models. Set the summary range appropriately depending on how long you want to keep the data, > 1 Month. The data model is updated every 10 minutes in order for the sessions to get picked up properly. The data model acceleration needs to finish before you will see any data in any dashboard except the "Real-Time" dashboard which uses raw log data as source. That means that you initially might not see data until the data model has finished building. This could initially take many hours depending on how much data it is trying to build.
If Events are showing 0 after install or upgrade you might have to rebuild the data model.
If you want to monitor certain browsing paths or pageviews you can configure goals. This is used if you for instance want to get conversion rates or funnel abandonment rates. You can find the Goals setup page under Goals->Goals Setup.
The goals are stored in a summary index called "goal_summary".
When enabling goals, the app will start monitor goal completions from the time you save the goal. To backfill goals there is a search called "Generate Goal summary - Backfill" which can be found under the Goals menu. Please note that running this search multiple times will mean the goal completions will be duplicated. To reset the goals you need to clean the "goal_summary" index.
The app uses a third party user agent parsing library that gets updated regularly. If you want to manually update these definitions you can download a yaml file here:
https://github.com/ua-parser/uap-core/blob/master/regexes.yaml
Copy the file "regexes.yaml" the folder
SplunkAppForWebAnalytics/bin/ua_parser
This update should take affect immediately.
THIS VERSION CONTAINS A BREAKING CHANGE. SEE BELOW.
From version 2.2.0, the app now uses the KV store to store the wesite configuration (collection WA_settings). This makes it a lot easier to edit and setup the website configuration step. This is something that many users have struggled with so I hope this change will make it easier during the initial setup.
If you are upgrading from an old version you need to take the mandatory step of migrating the old configuration from in the csv lookup to the KV store. This is done by going to the webite setup page and clicking the "Migrate old config" button. You only need to do this once.
Version 2.0.0 of the app has made a small change to the Web datamodel to increase compatibility with more Splunk versions. This might trigger a data model rebuild when upgrading. If you want to prevent this from happening, use the old data model definition file "web.conf" and delete the one provided wby this app. It's recommended to use the new version of web.conf as this is the version that will be used moving forwards.
The User Journey Flow dashboard now uses the official Sankey vizualisation add-on that needs to be downloaded separately in order for this dashboard to work. You can find this add-on here: https://splunkbase.splunk.com/app/3112/
The goal_summary index is now not created by the app. You need to manually add this index if you are using this feature in the app. All old data will be retained even if the index is not created by the app, just create the index manually and it will work.
There is a new dashboard, "Response Times" which help you find the slowest resources on your site. Your web server might not output the response time in the log by default so this needs to be enabled in order to make this dashboard work. On IIS this is often pre-enabled but on Apache and NGINX you need to add %D to the end of the log format settings of the server. More details on how to enabled this here:
http://httpd.apache.org/docs/current/mod/mod_log_config.html
If you add the %D to the end of the log format for the access_combined sourcetype, the field extractions will work by default.
Version 1.6 of the app uses the KV Store for the session lookups instead of a CSV file. This feature will only work on Splunk version 6.3 and above.
For 6.2 support of the app, you need to continue using a CSV file for the lookup.
To enable this you should replace the contents of, or the file:
"SPLUNK_HOME/etc/apps/SplunkAppForWebAnalytics/default/transforms.conf"
with the corresponding 6.2 compatible file that can be found under
"SPLUNK_HOME/etc/apps/SplunkAppForWebAnalytics/install/transforms_6.2.conf".
Restart Splunk after this is done.
Splunk Answers thread on Splunk App for Web Analytics.
https://answers.splunk.com/app/questions/2699.html
A lot of the problems customers have with the app have already been solved.
In the context of the app, try and do the search for:
tag=web
Based on the output of this search check the following
No data returned
If this is not returning any results I suspect you are not seeing the data because it is stored in a non-default index and the user in Splunk does not search in non-default indexes automatically. Another issue might be that you are not using any of the pre-configured sourcetypes. See Setup point 1 above.
Site field not present
If this is returning results, double check that each entry has the "site" field populated. It's crucial that this field exists in your data. See Setup point 2 above.
File field not present
Another field that is known to cause problems is the "file" field. This needs to be present in your field extractions and if it is not, you will not see the "eventtype-=pageview" which is necessary for the app to work. Make sure this is extracted correctly.
As the app relies heavily on data model accelerations you will not see anything in any dashboards (except the "Real-Time" ones) until this acceleration has completed. Initially this could take a while. There is a "Data Model Audit" dashboard that will tell you if the acceleration is complete or not.
The user agent parsing is based on an add-on developed by David Shpritz (TA-user-agents) who in turn uses a Python module from:
https://github.com/tobie/ua-parser
This version will trigger a datamodel rebuild as the name of the datamodel has changed. If you want to keep your existing datamodel name, you can update the macro called "datamodel" to point to your custom name.
Fix for Splunk Cloud 8.1
Modified eventtype web-traffic to include an index definition to make it easier to limit the app to just the relevant indexes.
Modified tag definition for the datamodel to be tag=webanalytics
Fix for Python3 and 8.1 Compatibility
Updated User Agent regex library
Fix for missing schedule for the Session saved searches
Updated the User Agent detecion library
Added support for Oracle Weblogic (sourcetype "oracle:weblogic")
Updated user agent library
NOTE: This upgrade contains a breaking change, you need to migrate your existing website settings from the WA_settings.csv file to the KV Store. This is done by clicking a button on the Sites setup dashboard at the bottom. It takes approximately 10 seconds and only needs to be done once.
Added support for the sourcetypes "apache:access" and "aws:cloudfront:accesslogs"
Added new panel "Session Time Distribution" on the Behavior dashboard
Fixed bugs in the comparison feature for the timecharts
Updated the user agent string parsing
As a Splunkbase app developer, you will have access to all Splunk development resources and receive a 10GB license to build an app that will help solve use cases for customers all over the world. Splunkbase has 1000+ apps from Splunk, our partners and our community. Find an app for most any data source and user need, or simply create your own with help from our developer portal.