Conversation
Add observations table logging each URL sighting with source, honeynet, session and timestamp; dedupe via NULL-normalized generated key with UNIQUE index so INSERT OR IGNORE is idempotent. process_new_session() takes an optional honeynet param and inserts one observation per URL; Warden, T-Pot, Honeynet.Asia and nested-URL collectors record their source and honeynet. discovered_urls/url_source left unchanged, provenance derived by joining observations.
softlukas
marked this pull request as draft
September 22, 2026 16:18
added 18 commits
September 23, 2026 23:13
migrate.sh backs up the active DB to <db_dir>/backup, then applies pending install/migrations/*.sql in order, tracking applied files in schema_migrations so each runs once. Moves the observations migration to install/migrations/001_observations.sql.
Add tabs per Figma (light theme). Sources aggregates observations per honeynet with an internal derived-from link; Overview keeps existing content; detail() computes the aggregation and badges.
…Class. History tab
…00:00:00 placeholder time
…nt dedup and change detection During the daily re-poll, capture fetch metadata and response body, deduplicate content via common.content_store, and write a download_observation row that records whether the content is new, unchanged, or changed. Add worker docstring and a comment explaining the URL chunking.
…cord fetch metadata Extend analyze_content so each HTTP fetch records a download_observation row (timestamp, source IP, status code, response headers) and usable payloads are persisted deduplicated on disk keyed by SHA-256 (via common.content_store). Adds _record_fetch/_store_downloaded_content helpers, VT quota global comments, and updates the analyze_content/evaluate_url docstrings.
…ns schema Add `content` (one row per unique SHA-256) and `download_observations` (one row per fetch) tables to create_db.sql for fresh installs, plus migration 003_content_storage.sql for existing DBs (purely additive, IF NOT EXISTS).
…oute Content tab groups identical consecutive downloads into versions (fetch count, source IP, SHA-256, size, headers, changed badge). Adds a /content/download/<id> route plus human_bytes/fmt_ts filters and tab styling.
…nload observations
Add sandbox_submissions table to create_db.sql tracking content samples sent to a malware sandbox (content_id FK, sha256, sandbox name, submitted_at/by, status, verdict, report_url to the external analysis report). Also add migration script.
log if skip is due to Body is None, or unable to import content_storage module and it is None
softlukas
force-pushed
the
feat-content-storage-and-sandbox
branch
from
September 28, 2026 13:34
cfe2e11 to
91e262f
Compare
added 4 commits
September 29, 2026 20:09
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.