Monitoring and Alerting with SGT. and StreamAlert - Part 1

SGT is focused on managing Osquery endpoints and collecting data.

StreamAlert is focused on ingesting data, processing and alerting.

Together, these two systems form an amazingly powerful monitoring and alerting stack which can scale from tiny environments from a few systems, to thousands of endpoints across an entire company’s production and corporate infrastructure.

In this series of posts, I’ll walk through how to set both projects up, gather some data from our endpoints and write some alerts to notify when certain events are triggered.

Note: While both systems could technically be deployed concurrently, the sake of clarity we will configure and deploy them one at a time

SGT

Because SGT is producing the data, we’ll set that up first.

Setup

prereqs:
  1. An AWS account with admin access to DynamoDB, EC2, ES (ElastisearchService), Kinesis/Firehose and IAM. (note, this must be programatic access, so you can have an access key and secret to use)
  2. Golang 1.8.2+
  3. Terraform 11.0+
  4. A domain with DNS managed via Route53 (Note: This does not mean you need to buy a domain, you can use an existing domain and just manage DNS on Route53)
  5. An SSL cert with public and private keypair. This will be used to terminate TLS connections to our server see Obtaining a free ssl cert for SGT with Letsencrypt for one method of aquiring a certificate
  6. An aws profile configured.

Installation

  1. Clone the repo

        
    go get github.com/OktaSecurityLabs/sgt
    
  2. change into the downloaded directory

    cd $GOPATH/src/github.com/OktaSecurityLabs/sgt
    
  3. Build the project

    go build
    
  4. Copy your ssl certs to the proper directory. For this blog, I’m using a subdomain of securelyinsecure.com with a letsencrypt certificate, sgt-demo.securelyinsecure.com. Lets encrypt certs live in /etc/letsencrypt/live/<site> so I’m copying them from there into the cert directory for SGT.

    sudo cp /etc/letsencrypt/live/sgt-demo.securelyinsecure.com/fullchain.pem certs/fullchain.pem
    sudo cp /etc/letsencrypt/live/sgt-demo.securelyinsecure.com/privkey.pem certs/privkey.pem
    
  5. Rename your certs to reflect which site they belong to. I recommend following the example format of

    example.domain.com.fullchain.pem
    

    moving…

    cd certs
    mv fullchain.pem sgt-demo.securelyinsecure.com.fullchain.pem
    mv privkey.pem sgt-demo.securelyinsecure.com.privkey.pem
    cd ..
    
  6. Create a new environment by following the prompts

    ./sgt wizard
    

    6a. Enter a name for your environment (I’m calling my demo one sgt-demo)

    Enter new environment name.  This is typically something like'Dev' or 'Prod' or 'Testing, but can be anything you want it to be: sgt-demo
    

    6b. Choose the AWS profile to use (Mine is again called sgt-demo)

    Enter the name for the aws profile you'd like to use to deploy this environment 
    if you've never created a profile before, you can read more about how to do this here
    http://docs.aws.amazon.com/cli/latest/userguide/cli-multiple-profiles.html 
    a 'default' profile is created if you've installed and configured the aws cli: 
    sgt-demo
    

    6c. Enter the IP address that you are currently deploying from.

    Enter an ipaddress or cidr block for access to your elasticsearch cluster. 
    Note:  This should probably be your current IP address, as you will need to be able to access 
    elasticsearch via API to create the proper indices and mappings when deploying: xxx.xxx.xxx.xxx/24
    

    6d. Name your log bucket. I recommend something easily identified for your domain.

    Enter a name for the s3 bucket that will hold your osquery logs. 
    Remeber, S3 bucket names must be globally unique: sgt-demo.log.bucket
    

    6e. And your config bucket…

    Enter a name for the s3 bucket that will hold your server configuration 
    Remember, S3 bucket names must be globally unique: 
    sgt-demo.configuration.bucket
    

    6f. Enter your root domain

    Enter the domain you will be using for your SGT server. 
    Note:  This MUST be a domain which you have previously registered or are managing throughaws. 
    This will be used to create a subdomain for the SGT TLS endpoint
    securelyinsecure.com
    

    6g. Enter the subdomain (sgt-demo in my case)

    Enter a subdomain to use as the endpoint.  This will be prepended to the 
    domain you provided as a subdomain
    sgt-demo
    

    6h. Enter your aws keypair name

    Enter the name of your aws keypair.  This is used to access ec2 instances ifthe need 
    should ever arise (it shouldn't).
    NOTE:  This is the name of the keypair EXCLUDING the .pem flie name and it must already exist in aws
    my-secret-key-name
    

    6i. Enter the name of your keypair and priv key, as you named them above.

    Enter the name of the full ssl certificate chain bundle you will be using for 
    your SGT server.  EG - full_chain.pem : 
    sgt-demo.securelyinsecure.com.fullchain.pem
    Enter the name of the private key for your ssl certificate.  Eg - privkey.pem:
    sgt-demo.securelyinsecure.com.privkey.pem
    

    6j. Enter the node secret

    Enter the node secret you will use to enroll your endpoints with the SGT server
    This secret will be used by each endpoint to authenticate to your server: 
    my-super-secret-node-secret
    

    6k. Enter the app secret

    Enter the app secret key which will be used to generate session tokens when 
    interacting with the API as an authenticated end-user.  Make this long, random and complex: 
    diu3piqeujr302348u33rqwu934r1@#)(*@3
    

    Select N when prompted to continue. Because this is a demo environment, we’re going to make a small change to our configuration.

  7. Edit the environment config file found in /terraform/<environment/environment.json with your favorite editor and change the value for create_elasticsearch to 0. This will disable the creation of elasticsearch, which we will not be using for this demo. In a production environment, Elasticsearch would be a large part of your process, but it adds significant cost and it’s not needed for this demo.

    {
      "environment": "example_environment",
      "aws_profile": "default",
      "user_ip_address": "127.0.0.1",
      "sgt_osquery_results_bucket_name": "example_log_bucket_name",
      "sgt_config_bucket_name": "example_config_bucket_name",
      "domain": "somedomain.com",
      "subdomain": "mysubdomain",
      "aws_keypair": "my_aws_ec2_keypair_name",
      "full_ssl_certchain": "full_cert_chain.pem",
      "ssl_private_key": "privkey.pem",
      "sgt_node_secret": "super_sekret_node_enrollment_key",
      "sgt_app_secret": "ultra_mega_sekret_key_you'll_never_give_to_anyone_not_even_your_mother",
      "create_elasticsearch": 0
    }
    

Deploy!!

Its finally time to deploy, although hopefully that wasn’t too painful. Deployment is by far the easiest part.

./sgt deploy -env <your environment name> -all

This will stand up the entire environment, including endpoint configuration scripts which we will use to set up some osquery nodes later. The entire process should take about 5-10 minutes depending on your internet connection, at which point you should be ready to install osquery on an endpoint and start receiving logs!