Skip to content

Commit 414ddd4

Browse files
committed
Initial v0.1.0 commit
Add Scout Query extension with script caching and quick execution tables Added some sample scripts to the content server for testing
1 parent f1f09b8 commit 414ddd4

32 files changed

+2434
-0
lines changed

.gitattributes

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
# Auto detect text files and perform LF normalization
2+
* text=auto

.gitignore

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
*/venv/
2+
*.pem
3+
.DS_Store

README.md

Lines changed: 69 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,69 @@
1+
# Scout Osquery Extension
2+
3+
**Scout** is an osquery extension that provides two powerful tables for executing and managing scripts in a secure and efficient manner.
4+
5+
## Tables
6+
7+
### 1. `scout_exec`
8+
The `scout_exec` table allows you to run hosted scripts immediately during query time. This is particularly useful for **Live Query** scenarios, where immediate execution and feedback are necessary. In future versions, this table will adopt a pub/sub execution model, allowing for asynchronous or longer-running processes to be handled in a more scalable way.
9+
10+
### 2. `scout_cache`
11+
The `scout_cache` table provides visibility into scripts cached on the endpoint. By caching scripts, Scout minimizes redundant network requests and optimizes resource usage. You can query this table to check whether an endpoint already has the script you need, ensuring faster and more efficient script execution.
12+
13+
## Security
14+
15+
To ensure security, **all scripts must be signed**. The osquery extension is configured with a public key to verify the integrity and authenticity of the scripts before execution. This guarantees that only trusted and verified scripts can be run on your endpoints.
16+
17+
## Running the Extension
18+
19+
### Step 1: Compile the Extension
20+
Once you have compiled the Scout extension, configure it by adding a `scout` block to your osquery configuration file. You will need to specify the URL for your script server and provide a public key for script verification.
21+
22+
- **`server_url`**: The URL where Scout will fetch scripts from.
23+
- **`public_key`**: The public key used to verify the integrity of the scripts.
24+
- **`cache_window`**: Optional - Duration for which the scripts are cached.
25+
- **`exec_timeout`**: Optional - Timeout for script execution.
26+
- **`cache_dir`**: Optional - Directory for caching scripts.
27+
28+
```json
29+
"scout": {
30+
"server_url": "http://localhost:5000/scripts",
31+
"public_key": "-----BEGIN PUBLIC KEY-----\n...Your New Public Key...\n-----END PUBLIC KEY-----",
32+
"cache_window": 3600,
33+
"exec_timeout": 60,
34+
"cache_dir": "/path/to/cache"
35+
}
36+
```
37+
38+
### Step 2: Start a Content Server
39+
You can use any web server to host signed scripts. For testing purposes, a simple content server and some example scripts are provided in the `content_server` directory.
40+
41+
To start the server:
42+
43+
1. Go to the `content_server` directory.
44+
2. Run your preferred web server (e.g., Python’s built-in server):
45+
46+
```bash
47+
cd content_server
48+
pip install -r requirements.txt
49+
python script_server.py
50+
```
51+
52+
Place any scripts you want to serve to clients in the scripts directory.
53+
54+
### Step 3: Run the Extension with Osquery
55+
56+
After setting up the server, you can run osquery with the Scout Query extension to start executing and caching scripts.
57+
58+
```bash
59+
./build.sh
60+
osqueryi --extension bin/scout-query-darwin-arm64.ext --allow-unsafe
61+
```
62+
63+
### Future Development
64+
65+
Scout Query is still in early development. The aim of this project is to allow for quick retrieval of data in a similar table format to existing osquery tables, securely distribute scripts around the network, and safely execute them. It includes some optional configs to limit the risk of long-running processes and denial of service on the download side.
66+
67+
There is a level of caching that takes place at the endpoint with signature verification and checks should the hosted content change or someone tamper with the local cached files.
68+
69+
We’re open to collaboration and would love to explore new use cases. If you find the Scout Query helpful or have any suggestions, feel free to reach out!

bin/README.md

Whitespace-only changes.

build.sh

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
#!/bin/bash
2+
3+
OUTPUT_DIR="../bin"
4+
SCOUT_DIR="scout-query"
5+
6+
cd $SCOUT_DIR || { echo "Directory $SCOUT_DIR not found. Aborting."; exit 1; }
7+
8+
platforms=("windows/amd64" "darwin/amd64" "darwin/arm64" "linux/amd64" "linux/arm64")
9+
10+
for platform in "${platforms[@]}"
11+
do
12+
platform_split=(${platform//\// })
13+
GOOS=${platform_split[0]}
14+
GOARCH=${platform_split[1]}
15+
output_name=$OUTPUT_DIR/scout-query-$GOOS-$GOARCH
16+
17+
if [ "$GOOS" == "windows" ]; then
18+
output_name+='.ext.exe'
19+
else
20+
output_name+='.ext'
21+
fi
22+
23+
env GOOS=$GOOS GOARCH=$GOARCH go build -ldflags "-w -X main.Version=0.1.0" -o $output_name 2> build_error.log
24+
if [ $? -ne 0 ]; then
25+
echo "An error occurred while building for $GOOS/$GOARCH. Aborting..."
26+
cat build_error.log
27+
exit 1
28+
fi
29+
done
30+
31+
rm -f build_error.log

content_server/app.py

Lines changed: 190 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,190 @@
1+
from flask import Flask, make_response, abort, jsonify
2+
import os
3+
import requests
4+
import hashlib
5+
from cryptography.hazmat.primitives import serialization, hashes
6+
from cryptography.hazmat.primitives.asymmetric import padding
7+
from flask_caching import Cache
8+
9+
# Configuration
10+
USE_GITHUB = os.environ.get('USE_GITHUB', '0') == '1'
11+
GITHUB_RAW_BASE_URL = "https://raw.githubusercontent.com/huntbase-io/scout-content/main"
12+
ALLOWED_OS_DIRS = ["windows", "linux", "darwin"]
13+
14+
# Local directories (used when USE_GITHUB=0)
15+
SCRIPTS_DIR = os.path.join(os.getcwd(), 'scripts')
16+
BIN_DIR = os.path.join(os.getcwd(), 'bin')
17+
18+
app = Flask(__name__)
19+
20+
# Configure caching (mainly for GitHub fetching)
21+
app.config['CACHE_TYPE'] = 'SimpleCache' # for production consider Redis or Memcached
22+
app.config['CACHE_DEFAULT_TIMEOUT'] = 300
23+
cache = Cache(app)
24+
25+
# Load the private key
26+
with open('private_key.pem', 'rb') as f:
27+
private_key = serialization.load_pem_private_key(
28+
f.read(),
29+
password=None,
30+
)
31+
32+
33+
# ---------- Helper Functions ----------
34+
@cache.memoize(timeout=300)
35+
def fetch_from_github(path):
36+
"""
37+
Fetch content from GitHub raw URL. Returns content or None if not found.
38+
"""
39+
url = f"{GITHUB_RAW_BASE_URL}/{path}"
40+
response = requests.get(url)
41+
if response.status_code == 200:
42+
return response.content
43+
return None
44+
45+
46+
def get_local_file_content(root_dir, filename):
47+
"""
48+
Get content from a local file. Returns content or None if not found.
49+
"""
50+
file_path = os.path.join(root_dir, filename)
51+
if not os.path.isfile(file_path):
52+
return None
53+
with open(file_path, 'rb') as f:
54+
return f.read()
55+
56+
57+
def get_script_content(os_dir, filename):
58+
"""
59+
Get script content either from GitHub or locally, depending on USE_GITHUB.
60+
If GitHub fails or USE_GITHUB=0, fallback to local if desired.
61+
"""
62+
# Validate OS directory if provided
63+
if os_dir and os_dir not in ALLOWED_OS_DIRS:
64+
return None
65+
66+
# Construct path
67+
if os_dir:
68+
path = f"scripts/{os_dir}/{filename}"
69+
local_path = os.path.join(SCRIPTS_DIR, os_dir, filename)
70+
else:
71+
path = f"scripts/{filename}"
72+
local_path = os.path.join(SCRIPTS_DIR, filename)
73+
74+
content = None
75+
if USE_GITHUB:
76+
# Try to fetch from GitHub
77+
content = fetch_from_github(path)
78+
# If not found on GitHub, we could optionally fallback to local:
79+
# if content is None:
80+
# content = get_local_file_content(os.path.join(SCRIPTS_DIR, os_dir) if os_dir else SCRIPTS_DIR, filename)
81+
else:
82+
# Local only
83+
content = get_local_file_content(os.path.join(SCRIPTS_DIR, os_dir) if os_dir else SCRIPTS_DIR, filename)
84+
85+
return content
86+
87+
88+
def get_bin_content(filename):
89+
"""
90+
Get binary content either from GitHub or locally.
91+
"""
92+
path = f"bin/{filename}"
93+
if USE_GITHUB:
94+
content = fetch_from_github(path)
95+
# If desired, fallback to local if GitHub not found:
96+
# if content is None:
97+
# content = get_local_file_content(BIN_DIR, filename)
98+
else:
99+
content = get_local_file_content(BIN_DIR, filename)
100+
return content
101+
102+
103+
def sign_content(content):
104+
"""
105+
Sign the given content using the private key.
106+
"""
107+
signature = private_key.sign(
108+
content,
109+
padding.PKCS1v15(),
110+
hashes.SHA256()
111+
)
112+
return signature.hex()
113+
114+
def compute_hash(content):
115+
"""
116+
Compute SHA256 hash of the content.
117+
"""
118+
hasher = hashlib.sha256()
119+
hasher.update(content)
120+
return hasher.hexdigest()
121+
122+
123+
# ---------- Routes ----------
124+
125+
@app.route('/bin/<path:filename>', methods=['GET'])
126+
def serve_bin(filename):
127+
content = get_bin_content(filename)
128+
if content is None:
129+
abort(404, description="Binary not found")
130+
131+
signature_hex = sign_content(content)
132+
response = make_response(content)
133+
response.headers['Content-Type'] = 'application/octet-stream'
134+
response.headers['X-Signature'] = signature_hex
135+
return response
136+
137+
138+
@app.route('/scripts/<path:filename>', methods=['GET'])
139+
def serve_script_no_os(filename):
140+
# This route is for scripts without OS directory specification
141+
content = get_script_content(None, filename)
142+
if content is None:
143+
abort(404, description="Script not found")
144+
145+
signature_hex = sign_content(content)
146+
response = make_response(content)
147+
response.headers['Content-Type'] = 'application/octet-stream'
148+
response.headers['X-Signature'] = signature_hex
149+
return response
150+
151+
152+
@app.route('/scripts/<os_dir>/<path:filename>', methods=['GET'])
153+
def serve_script(os_dir, filename):
154+
# Validate OS directory and fetch the script
155+
content = get_script_content(os_dir, filename)
156+
if content is None:
157+
abort(404, description="Script not found")
158+
159+
signature_hex = sign_content(content)
160+
response = make_response(content)
161+
response.headers['Content-Type'] = 'application/octet-stream'
162+
response.headers['X-Signature'] = signature_hex
163+
return response
164+
165+
166+
@app.route('/scripts/hash/<path:filename>', methods=['GET'])
167+
def get_script_hash_no_os(filename):
168+
content = get_script_content(None, filename)
169+
if content is None:
170+
abort(404, description="Script not found")
171+
172+
script_hash = compute_hash(content)
173+
return jsonify({"script_hash": script_hash})
174+
175+
176+
@app.route('/scripts/hash/<os_dir>/<path:filename>', methods=['GET'])
177+
def get_script_hash(os_dir, filename):
178+
content = get_script_content(os_dir, filename)
179+
if content is None:
180+
abort(404, description="Script not found")
181+
182+
script_hash = compute_hash(content)
183+
return jsonify({"script_hash": script_hash})
184+
185+
186+
if __name__ == '__main__':
187+
# Run the app
188+
# Set the USE_GITHUB=1 environment variable before running if you want GitHub fetching
189+
# Example: USE_GITHUB=1 python unified_app.py
190+
app.run(host='127.0.0.1', port=5000)

content_server/requirements.txt

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
cryptography==44.0.0
2+
Flask==3.1.0
3+
Flask_Caching==2.3.0
4+
psutil==6.1.1
5+
Requests==2.32.3
Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
#!/bin/bash
2+
3+
# Ensure the script is run as root (mimicking the given example's behavior)
4+
if [[ $EUID -ne 0 ]]; then
5+
echo "This script must be run as root"
6+
exit 1
7+
fi
8+
9+
# Set the directory for the default Chrome profile
10+
EXT_DIR="$HOME/Library/Application Support/Google/Chrome/Default/Extensions"
11+
12+
# Ensure jq is installed
13+
if ! command -v jq &> /dev/null; then
14+
echo "jq is not installed. Please install it (e.g., brew install jq) and run again."
15+
exit 1
16+
fi
17+
18+
# Check if the Chrome Extensions directory exists
19+
if [ ! -d "$EXT_DIR" ]; then
20+
echo "Chrome Extensions directory not found at: $EXT_DIR"
21+
exit 1
22+
fi
23+
24+
# Iterate over each extension directory
25+
for EXT_ID in "$EXT_DIR"/*; do
26+
if [ -d "$EXT_ID" ]; then
27+
EXT_ID_NAME=$(basename "$EXT_ID")
28+
29+
# Find the latest version directory by sorting versions and selecting the last
30+
LATEST_VERSION_DIR=$(ls "$EXT_ID" | sort -V | tail -n 1)
31+
MANIFEST="$EXT_ID/$LATEST_VERSION_DIR/manifest.json"
32+
33+
if [ -f "$MANIFEST" ]; then
34+
# Extract name and version from manifest.json
35+
NAME=$(jq -r '.name // "unknown_name"' "$MANIFEST")
36+
VERSION=$(jq -r '.version // "unknown_version"' "$MANIFEST")
37+
else
38+
NAME="unknown_name"
39+
VERSION="unknown_version"
40+
fi
41+
42+
# Print the extension info in a single row with lowercase field names and underscores
43+
echo -e "\"name\": \"$NAME\", \"version\": \"$VERSION\", \"extension_id\": \"$EXT_ID_NAME\""
44+
fi
45+
done
Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
#!/bin/bash
2+
3+
# Ensure the script is run as root
4+
if [[ $EUID -ne 0 ]]; then
5+
echo "This script must be run as root"
6+
exit 1
7+
fi
8+
9+
# Print column names
10+
echo -e "\"device\", \"link_speed\""
11+
12+
# Read the output of networksetup and process each Hardware Port block
13+
networksetup -listallhardwareports | while IFS= read -r line; do
14+
if [[ "$line" == "Hardware Port:"* ]]; then
15+
# Extract Hardware Port name (optional, not used in this format)
16+
hardware_port=$(echo "$line" | awk -F": " '{print $2}')
17+
elif [[ "$line" == "Device:"* ]]; then
18+
# Extract Device identifier (e.g., en0, en1)
19+
device=$(echo "$line" | awk -F": " '{print $2}')
20+
21+
# Retrieve the media status for the device
22+
link_speed=$(networksetup -getMedia "$device" 2>/dev/null | awk -F": " '{print $2}' | tr '\n' ' ' | sed 's/ $//')
23+
24+
# If link_speed is empty, set it to "unknown"
25+
if [ -z "$link_speed" ]; then
26+
link_speed="unknown"
27+
fi
28+
29+
# Output the device and link speed in a single row with lowercase field names and underscores
30+
echo -e "\"device\": \"$device\", \"link_speed\": \"$link_speed\""
31+
fi
32+
done

0 commit comments

Comments
 (0)