Part 3: Augment your SIEM

Scanner is often used to augment traditional SIEMs and log search tools.

Here are some common use cases:

Gain more visibility. Scanner can search archived historical logs in S3 at high speed (and low cost). This gives users much more visibility into large data sets stashed away in their archives.

Reduce costs. Teams with a high volume of data (i.e. hundreds of GB per day or more) can reduce costs by $500k to $1M per year by migrating a small number of log sources from a traditional SIEM to Scanner.

In this part of the guide, we'll look at a few examples showing how Scanner can enhance existing tools.

Scanner for Splunk

For teams using Splunk, you can install the Scanner for Splunk app available in Splunkbase. It's compatible with both Splunk Enterprise and Splunk Cloud.

The Scanner for Splunk app will introduce two new custom search commands you can use: scanner and scannertable. These commands will execute queries against the Scanner API, allowing you to view and manipulate the search results directly in Splunk.

Here is how you can get started.

Set up API key in Scanner

In Scanner, create an API key. Go to Settings > API Keys and click Create API Key. Give it a name, create it, and assign a role to it.

You can use the admin role with this API key, which is fine for the purposes of experimenting with the playground environment.

You can also use the default user role with this API key, but make sure to edit the user role such that it has permission to read and query the main index where the playground data is housed.

Set up the custom app in Splunk

In your Splunk environment, install Scanner for Splunk from Splunkbase.

Open up the Scanner for Splunk configuration page in Splunk. It should appear in your side bar, or you can find it by viewing the list of installed apps.

In the configuration page in Splunk:

• Copy-paste the Scanner API URL, which is available in Settings > API Keys in Scanner.

• Add a new API key. Choose the Splunk role to use with that API key. Copy-paste your API key from Scanner into here.

Whenever that role uses a Scanner custom search command, the role will use that API key.

For more information, see the Scanner for Splunk > Getting Started documentation.

Run Scanner search queries in Splunk

In Splunk, go to Search & Reporting.

Run this scanner query to view recent logs for a few users:

| scanner q="userIdentity.userName=(pgibbons samirn mbolton)"

In the search results, you should see log events returned from the playground environment.

Run this scannertable query to summarize the AWS activity for these users:

| scannertable q="userIdentity.userName=(pgibbons samirn mbolton)
| groupbycount eventSource, eventName, userIdentity.userName"

You can use scanner and scannertable commands in reports, dashboards, and many other places in Splunk.

Teams tend to use Scanner for Splunk for two use cases:

• Reduce Splunk costs. High volume log sources can be expensive to ingest into Splunk. Teams can reduce Splunk costs by 6 or 7 figures by moving a few high volume log sources into S3. Using Scanner, they can continue to search these logs from Splunk.

• Gain visibility into archived logs in S3. For teams who already have historical archives in S3, Scanner allows them to search these archives directly from Splunk. This can be helpful for investigations that need to examine long-term historical data, like hunting for advanced persistent threats.

Scanner for Grafana

For teams using Grafana, you can install the Scanner Grafana plugin available in Grafana plugins listing.

Set up API key in Scanner

In Scanner, create an API key. Go to Settings > API Keys and click Create API Key. Give it a name, create it, and assign a role to it.

You can use the admin role with this API key, which is fine for the purposes of experimenting with the playground environment.

You can also use the default user role with this API key, but make sure to edit the user role such that it has permission to read and query the main index where the playground data is housed.

Set up a Scanner data source in Grafana

Follow the Scanner for Grafana > Getting Started documentation to set up the Scanner plugin in Grafana and configure a new data source.

Use the Scanner API URL and the API key from Settings > API Keys in Scanner.

Run ad-hoc queries in Grafana

Within Grafana, go to Explore, and select the Scanner data source you created in the step above.

You will be able to write ad-hoc queries and view log results.

Run this query to view recent logs for a few users:

userIdentity.userName=(pgibbons samirn mbolton)

In the search results, you should see log events returned from the playground environment.

Run this query to summarize the AWS activity for these users:

userIdentity.userName=(pgibbons samirn mbolton)
| groupbycount eventSource, eventName, userIdentity.userName

You can use these queries in bar charts and other dashboard widgets as you see fit.

Teams tend to use Scanner for Grafana for two use cases:

• Fast needle-in-haystack search. Most log search tools that use S3 for storage have trouble with needle-in-haystack search across multiple data sources. Typical needle-in-haystack searches for a security team tend to be looking for indicators of compromise, like IP addresses, domains, file hashes, etc. Thanks to Scanner's inverted index in S3, these searches are fast, even across large data sets and vast time ranges.

• Gain visibility into archived logs in S3. For teams who already have historical archives in S3, Scanner allows them to search these archives directly from Grafana. This can be helpful for investigations that need to examine long-term historical data, like hunting for advanced persistent threats.