SGT is focused on managing Osquery endpoints and collecting data.

StreamAlert is focused on ingesting data, processing and alerting.

Together, these two systems form an amazingly powerful monitoring and alerting stack which can scale from tiny environments from a few systems, to thousands of endpoints across an entire company’s production and corporate infrastructure.

In this series of posts, I’ll walk through how to set both projects up, gather some data from our endpoints and write some alerts to notify when certain events are triggered.

Note: While both systems could technically be deployed concurrently, the sake of clarity we will configure and deploy them one at a time

SGT

Because SGT is producing the data, we’ll set that up first.

Setup

prereqs:

1. An AWS account with admin access to DynamoDB, EC2, ES (ElastisearchService), Kinesis/Firehose and IAM. (note, this must be programatic access, so you can have an access key and secret to use)
2. Golang 1.8.2+
3. Terraform 11.0+
4. A domain with DNS managed via Route53 (Note: This does not mean you need to buy a domain, you can use an existing domain and just manage DNS on Route53)
5. An SSL cert with public and private keypair. This will be used to terminate TLS connections to our server see Obtaining a free ssl cert for SGT with Letsencrypt for one method of aquiring a certificate
6. An aws profile configured.

Installation

1. Clone the repo

       
   go get github.com/OktaSecurityLabs/sgt
   

2. change into the downloaded directory

   cd $GOPATH/src/github.com/OktaSecurityLabs/sgt
   

3. Build the project

   go build
   

4. Copy your ssl certs to the proper directory. For this blog, I’m using a subdomain of securelyinsecure.com with a letsencrypt certificate, sgt-demo.securelyinsecure.com. Lets encrypt certs live in /etc/letsencrypt/live/<site> so I’m copying them from there into the cert directory for SGT.

   sudo cp /etc/letsencrypt/live/sgt-demo.securelyinsecure.com/fullchain.pem certs/fullchain.pem
   sudo cp /etc/letsencrypt/live/sgt-demo.securelyinsecure.com/privkey.pem certs/privkey.pem
   

5. Rename your certs to reflect which site they belong to. I recommend following the example format of

   example.domain.com.fullchain.pem
   

   moving…

   cd certs
   mv fullchain.pem sgt-demo.securelyinsecure.com.fullchain.pem
   mv privkey.pem sgt-demo.securelyinsecure.com.privkey.pem
   cd ..
   

6. Create a new environment by following the prompts

   ./sgt wizard
   

   6a. Enter a name for your environment (I’m calling my demo one sgt-demo)

   Enter new environment name.  This is typically something like'Dev' or 'Prod' or 'Testing, but can be anything you want it to be: sgt-demo
   

   6b. Choose the AWS profile to use (Mine is again called sgt-demo)

   Enter the name for the aws profile you'd like to use to deploy this environment 
   if you've never created a profile before, you can read more about how to do this here
   http://docs.aws.amazon.com/cli/latest/userguide/cli-multiple-profiles.html 
   a 'default' profile is created if you've installed and configured the aws cli: 
   sgt-demo
   

   6c. Enter the IP address that you are currently deploying from.

   Enter an ipaddress or cidr block for access to your elasticsearch cluster. 
   Note:  This should probably be your current IP address, as you will need to be able to access 
   elasticsearch via API to create the proper indices and mappings when deploying: xxx.xxx.xxx.xxx/24
   

   6d. Name your log bucket. I recommend something easily identified for your domain.

   Enter a name for the s3 bucket that will hold your osquery logs. 
   Remeber, S3 bucket names must be globally unique: sgt-demo.log.bucket
   

   6e. And your config bucket…

   Enter a name for the s3 bucket that will hold your server configuration 
   Remember, S3 bucket names must be globally unique: 
   sgt-demo.configuration.bucket
   

   6f. Enter your root domain

   Enter the domain you will be using for your SGT server. 
   Note:  This MUST be a domain which you have previously registered or are managing throughaws. 
   This will be used to create a subdomain for the SGT TLS endpoint
   securelyinsecure.com
   

   6g. Enter the subdomain (sgt-demo in my case)

   Enter a subdomain to use as the endpoint.  This will be prepended to the 
   domain you provided as a subdomain
   sgt-demo
   

   6h. Enter your aws keypair name

   Enter the name of your aws keypair.  This is used to access ec2 instances ifthe need 
   should ever arise (it shouldn't).
   NOTE:  This is the name of the keypair EXCLUDING the .pem flie name and it must already exist in aws
   my-secret-key-name
   

   6i. Enter the name of your keypair and priv key, as you named them above.

   Enter the name of the full ssl certificate chain bundle you will be using for 
   your SGT server.  EG - full_chain.pem : 
   sgt-demo.securelyinsecure.com.fullchain.pem
   Enter the name of the private key for your ssl certificate.  Eg - privkey.pem:
   sgt-demo.securelyinsecure.com.privkey.pem
   

   6j. Enter the node secret

   Enter the node secret you will use to enroll your endpoints with the SGT server
   This secret will be used by each endpoint to authenticate to your server: 
   my-super-secret-node-secret
   

   6k. Enter the app secret

   Enter the app secret key which will be used to generate session tokens when 
   interacting with the API as an authenticated end-user.  Make this long, random and complex: 
   diu3piqeujr302348u33rqwu934r1@#)(*@3
   

   Select N when prompted to continue. Because this is a demo environment, we’re going to make a small change to our configuration.

7. Edit the environment config file found in /terraform/<environment/environment.json with your favorite editor and change the value for create_elasticsearch to 0. This will disable the creation of elasticsearch, which we will not be using for this demo. In a production environment, Elasticsearch would be a large part of your process, but it adds significant cost and it’s not needed for this demo.

   {
     "environment": "example_environment",
     "aws_profile": "default",
     "user_ip_address": "127.0.0.1",
     "sgt_osquery_results_bucket_name": "example_log_bucket_name",
     "sgt_config_bucket_name": "example_config_bucket_name",
     "domain": "somedomain.com",
     "subdomain": "mysubdomain",
     "aws_keypair": "my_aws_ec2_keypair_name",
     "full_ssl_certchain": "full_cert_chain.pem",
     "ssl_private_key": "privkey.pem",
     "sgt_node_secret": "super_sekret_node_enrollment_key",
     "sgt_app_secret": "ultra_mega_sekret_key_you'll_never_give_to_anyone_not_even_your_mother",
     "create_elasticsearch": 0
   }
   

Deploy!!

Its finally time to deploy, although hopefully that wasn’t too painful. Deployment is by far the easiest part.

./sgt deploy -env <your environment name> -all

This will stand up the entire environment, including endpoint configuration scripts which we will use to set up some osquery nodes later. The entire process should take about 5-10 minutes depending on your internet connection, at which point you should be ready to install osquery on an endpoint and start receiving logs!