Please consult the Splunk on Splunk User Manual and this Splunk Answer to learn about best practices to deploy the S.o.S app in a distributed environment.
1) If Sideview Utils version 1.1.7 (or later) is not installed, please install or update it before installing S.o.S.
Download Sideview Utils from Splunkbase
2) Install the S.o.S app:
If you have a distributed search environment, make sure you install S.o.S on the search-head(s) only. From the search-head the app can discover search-peers present in the distributed deployment.
3) Download and install the Splunk on Splunk add-on (S.o.S TA) on search peers to provide resource usage information to S.o.S.
This add-on provides data inputs that gather memory and CPU usage for Splunk Web, splunkd, and search processes as well as other system resource information. For more information, see the README file available with the S.o.S TAs.
Note: You do not need to install the S.o.S add-on on a Splunk instance were the S.o.S app is already installed. The S.o.S app ships with the same data inputs.
[SOS-11] Fixed an issue where ps_sos.ps1 would log many "WriteError" exceptions to splunkd.log and insert incorrect values in its events.
[SOS-12] Fixed an issue where the "Security Health Check" view would fail to show results on a Splunk Enterprise 6.2 instance.
[SOS-19] Retired the "Bucket information" panel in the "Cluster Master View" as it was dispatching potentially unsafe rest-based searches against the Cluster Master's buckets endpoint.
[SOS-39] The securityinfo.py search command - and by extension, the "Security Health Check" view - now appropriately scopes its results to the instance picked by the user.
[SOS-40] Fixed an issue where the "cluster" command would fail to show event cluster counts due to a change in internal behavior.
[SOS-113][SOS-117][SOS-141] Forwarder instances will no longer be listed in the "Host" pulldown of the "Search Usage Patterns, "Scheduler Activity" and "Search Activity" views.
Full support for Splunk Enterprise 6.1
NEW VIEW: Search > Search Activity
Provides deep insight into instance-scoped search workload, expressed as search concurrency, resource usage or aggregate search time. These metrics can be grouped by various relevant search properties: mode (historical vs. real-time), type (ad hoc vs. scheduled), user...
NEW VIEW: Resource Usage > Indexes Disk Usage and Properties
Allows a deployment-wide or instance-scoped view of index disk usage and other properties. Can be scoped to one or all indexes.
NEW VIEW: Deployment Status > Warnings and Errors > Security Health Check
A series of checks against security settings in your Splunk Enterprise installation.
NEW VIEW: Indexing > Index Replication > Cluster Service Activity
Shows service activity in a Cluster in great detail, allowing to better understand maintenance and repair operations undertaken by the Cluster Master and its peers.
24 bugs fixed! See the RELEASE-NOTES file for full details.
New features for the Deployment Topology view
Data overlays for instance status and resource usage (CPU/Memory).
NEW VIEW - Search > Search-head Pooling Performance
Check the usage and performance of the NFS shared storage device central to search-head pooling deployments. Compare performance metrics both at the storage (NFS) and application (Splunk) levels.
NEW VIEW - Indexing > Metrics > License Usage - Today
Get a license usage report for the current day and a history of license warnings for the current license window. (Applies to Splunk 4.3.x and 5.x only)
NEW VIEW - Indexing > Metrics > License Usage - Last 30 Days
Get a daily license usage report for the past 30 days and break it down by pool, indexer, source, sourcetype or host. (Applies to Splunk 4.3.x and 5.x only)
NEW VIEW - Indexing > Index Replication > Bucket Fix-up Activity
Monitor the status and progress of bucket fix-up operations in a cluster.
10 bugs fixed! See the README file for full details.
[SUP-723] Fixed an issue where scheduled searches "sos_splunk_instances_info" and "sos_refresh_splunk_servers_cache" would run several times per minute instead of at their scheduled time on a pooled search-head running Splunk 5.0.3. Note that the root cause of this problem is core Splunk bug SPL-68970.
[SUP-720] Fixed an issue where the Home view would be caught in a reload loop after S.o.S was installed or upgraded on a pooled search-head running Splunk 5.0.3.
[SUP-716] File $SPLUNK_HOME/var/log/splunk/sos_ftr.log is now explicitly sourcetyped.
[SUP-715] Our invocations of the "btool" command with the "--debug" flag no longer cause logs to be appended to $SPLUNK_HOME/var/log/splunk/btool.log.
[SUP-701] Fixed an issue where the Data Inputs > Tailing Processor view would fail to display when scoped to instances running Windows, showing instead an error banner stating "Invalid header received from stream generating script tpstatusquery".
[SUP-692] Fixed an issue where the in-product app browser wouldn't be scoped
to the Sideview Utils app during the installation workflow.
[SUP-668] There is now a scheduled search populating the "splunk_forwarders_cache.csv" lookup table with forwarder information.
[SUP-657] Added a spec file describing the "splunk_servers_cache.csv" lookup table.
[SUP-630] Created a macro to qualify searches based on their search ID.
[SUP-627] Fixed an issue where the ps_sos.sh scripted input would no longer print out full process arguments when executed by Splunk 5.x on Solaris.
[SUP-619] Metrics: Fixed an issue where the license usage chart would improperly show a "license_audit" pool for a license self-master.
[SUP-616] Fixed an issue with the ps_sos.ps1 scripted input where memory usage would sometimes be recorded as a negative value.
[SUP-596] Metrics: Fixed an issue where the license usage chart would not show multiple pools.
[SUP-578] Retired the "Distributed Searches Memory Usage" view.
[SUP-573] A new scripted input is now available to monitor the I/O usage of pooled search-heads on the shared NFS device: nfs-iostat_sos.py
[SUP-565] Fixed an issue where the ps_sos.ps1 scripted input would not run on an instance part of a search-head pool.
[SUP-541] Updated the app icon.
[SUP-540] Updated the app screenshot displayed on Splunkbase.
[SUP-530] Splunk File Descriptor Usage: The time stamp of the data sample used to populate the view is now shown.
[SUP-475] Dispatch Directory Inspector: Added a search box to filter results.
[SUP-474] Dispatch Directory Inspector: Added some statistical aggregations at the top of the view.
[SUP-606] Splunk CPU/Memory Usage: Resolved a problem where the memory usage charts would fail to report the memory usage of certain search processes.
[SUP-600] Metrics: Fixed an issue with the license reporting panel, which would show inaccurate numbers when multiple license pools are defined.
[SUP-599] Resolved a problem where the host "tag" for instances listed in the "Server to query" pulldown would not be properly determined on Splunk 5.x.
[SUP-595] Indexing Performance: Fixed an issue where no data points would be drawn when "Last 15mn" is selected from the time picker.
[SUP-589] Data Inputs Overview: Fixed an issue where this view would show no results when running on Splunk 5.x.
[SUP-587] Splunk CPU/Memory Usage: Renamed the "splunkd" series to "splunkd service".
[SUP-585] Metrics: Ensured that internal indexes and sourcetypes are no longer excluded from indexing volume reports.
[SUP-584] Metrics: Fixed an issue where excessive division for indexing volume metrics would lead to inaccurate reporting.
[SUP-583] Metrics: Fixed an issue where outgoing network throughput would be inaccurate by one order of magnitude when a split-by clause was used.
[SUP-582] Fixed an issue where an improper value for the "count" parameter of the "rest" command would cause a red error banner.
[SUP-558] Added an outputs.conf file with configuration that, if enabled, ensures that _internal events are forwarded from search-head to indexers.
[SUP-556] Fixed an issue where the "level" parameter of the Messages module would cause a red error banner on certain versions of Splunk.
[SUP-555] Resolved an issue where the "Server to query" pulldown on the Home view was not sorting hosts properly.
[SUP-554] Forwarders are now excluded by the searches of the Distributed Indexing view.
[SUP-547] Added a panel to the Indexing Performance view to expose subtask- level CPU time usage metrics for the indexer pipe which are new in 5.x.
[SUP-545] Adapted the searches against events generated by the ps_sos.* scripted inputs to the new splunkd process command line format in 5.x.
[SUP-527] Updated the build2version.csv lookup with information for the latest Splunk releases.
[SUP-538] Inputs Overview: Fixed a bug where the drilldown to file monitor input details would break due to a regular expression not supporting Windows paths.
[SUP-537] Home: Fixed a bug that caused the search powering the "A glimpse of your Splunk instance" panel to mismatch field values across hosts.
[SUP-532] Configuration File Comparator: General uncluttering and visual sanitization of this view.
[SUP-528] Distributed Indexing Performance: Set the height of the charts to a sensible default value.
[SUP-526] Scheduler Activity: Fixed wrong total execution count reported in the "Scheduler Activity" and "Execution Count by App/SavedSearch Name" panels.
[SUP-524] Scheduler Activity: Fixed a field extraction that was causing a NULL series to appear in the "Execution Count by App/SavedSearch Name" panel.
[SUP-521] Splunk CPU/Memory Resource Usage: Updated the search strings in the in-view help.
[SUP-507] Documented the search strings used for the Data Inputs Overview and Dispatch Directory Inspector in the in-view help.
[SUP-505] Fixed a typo in the lsof_sos.sh scripted input.
[SUP-503] Entries in the "Server to query" pulldown are now sorted based on the role of the Splunk instance: search-heads > search peers > forwarders.
[SUP-478] In the Errors view, improved chart readability by moving legends underneath the charting area.
2 bugs and 4 new features in this version! Check the CHANGELOG file for details.
New features for 2.0:
Centralized Splunk instance troubleshooting
Tracking Splunk resource usage
Improved searches and data representation
Improved help panels and troubleshooting documentation
Improved visual theme
As a Splunkbase app developer, you will have access to all Splunk development resources and receive a 10GB license to build an app that will help solve use cases for customers all over the world. Splunkbase has 1000+ apps from Splunk, our partners and our community. Find an app for most any data source and user need, or simply create your own with help from our developer portal.