cmd/seed: synthetic dataset for local development by ZukwiZ · Pull Request #70 · csfloat/reverse-watch

ZukwiZ · 2026-05-27T17:07:49Z

Summary

Adds a dev-only go run ./cmd/seed command that loads a synthetic ~6-month dataset (~2k rows, gentle sinusoid with occasional spikes and quiet days) into the local public database. The data always ends today, so every dashboard period (7d / 30d / 3m / 6m / 1y) has something to render without needing production data.

Safety properties

Local only. Refuses to run unless Environment is development and Database.Host is local (localhost, a loopback IP, a unix socket path, or a single-label docker service name such as postgres). Hostnames with a dot or colon, including Cloud SQL connection names, are rejected.
No real Steam accounts. Steam IDs come from the unallocated top of the 32-bit account space (76561201986797568–76561202255233023). They pass SteamID.IsValid but don't belong to anyone.
Doesn't touch the CSFloat ingestor. Marketplace slugs are dev- prefixed (dev-csfloat, dev-tradeit, ...) and rows have no reporter_internal_id, so the ingestor's resume cursor (highest reporter_internal_id on csfloat rows) is unaffected.
Safe to rerun. Inserts use ON CONFLICT (steam_id, marketplace_slug) WHERE deleted_at IS NULL DO NOTHING, so a plain rerun inserts nothing instead of erroring.
Refreshable. go run ./cmd/seed -reset hard-deletes every row in the synthetic Steam ID range first, then inserts a fresh dataset ending today. Real rows are never touched.

Other details

Snowflake IDs use the same created_at + sequence layout as domain/models/snowflake.go, so synthetic rows sort chronologically alongside real ones.
Today's rows are spread across the part of the day that has already passed, rather than clamped to a single timestamp.
Inserts run on a Warn-level gorm session, so the 1,000-row batches don't print SQL.
.gitignore adds internal-docs/ and internal/devseed/fixtures/.

Test plan

go test ./... against local Postgres 16
go run ./cmd/seed → 2006 inserted, 0 already present
Rerun → 0 inserted, 2006 already present, plus a hint to use -reset
go run ./cmd/seed -reset → deletes 2006 rows, inserts 2006 fresh rows
DATABASE_HOST=db.example.com and DATABASE_HOST=my-proj:us-central1:rw both exit non-zero with a refusal message
After seeding, the dashboard at / shows ~6 months of activity across all period ranges (needs static: public Reverse Watch dashboard #71)
Made with Cursor

Note

Low Risk
Dev-only CLI with an environment guard and idempotent inserts; not linked to the production server binary.

Overview
Adds a dev-only seed path so local Postgres can hold a deterministic ~6-month reversal history (~9.8k rows) without production exports.

go run ./cmd/seed loads config, exits unless Environment is development, then bulk-inserts generated rows into the public DB via ON CONFLICT (id) DO NOTHING (safe to re-run). internal/devseed builds daily volume with variance, marketplace mix, sources, optional expungements, synthetic Steam IDs, and snowflake IDs aligned with production ordering.

README documents the workflow and dashboard period coverage; .gitignore excludes internal-docs/ and optional internal/devseed/fixtures/.

^{Reviewed by Cursor Bugbot for commit b067b6b. Bugbot is set up for automated code reviews on this repo. Configure here.}

Add a deterministic ~6-month synthetic dataset (~9.8k rows, gentle sinusoid with occasional spikes and quiet days) for exercising the dashboard locally without needing real production exports. The generator deliberately spans every period (7d / 30d / 3m / 6m / 1y) so the chart UI has data to render at any range. Safety properties: - Refuses to run unless Config.Environment == "development". - INSERT … ON CONFLICT (id) DO NOTHING, so re-running is a no-op. - Steam IDs use a clearly-synthetic 76561198000000000 prefix. - Snowflake IDs encode the same created_at + sequence layout as the production generator, so synthetic rows sort chronologically alongside any real rows already in the DB. internal-docs/ and internal/devseed/fixtures/ are added to .gitignore to keep author scratch space and any future local CSV fixtures out of the public repo. Co-authored-by: Cursor <cursoragent@cursor.com>

zedimytch · 2026-06-22T20:50:57Z

+const (
+	syntheticRNGSeed      int64  = 42
+	syntheticDays                = 180
+	syntheticTargetTotal         = 9800


Specify the datatype here.
Also, we probably don't need 9800 entries, a couple thousand should work.

Done — syntheticTargetTotal int = 2000.

zedimytch · 2026-06-22T20:51:01Z

+// days). Snowflake IDs are unique within the slice and won't collide with
+// real CSV-seeded IDs, so callers can pipe the result straight into
+// InsertReversals.
+func GenerateSynthetic(now time.Time) []*models.Reversal {


Let's normalize now() to UTC before we derive any dates.

Done — now() is normalized to UTC before any dates are derived.

zedimytch · 2026-06-22T20:51:07Z

+The seed:
+
+- Refuses to run unless `Environment` is `development`.
+- Uses `INSERT … ON CONFLICT (id) DO NOTHING`, so it's safe to re-run.
+- Generates a deterministic 6-month dataset so the dashboard at `/` has enough data to exercise every period (7d / 30d / 3m / 6m / 1y).
+- Uses a synthetic Steam ID prefix (`76561198000000000`) so generated IDs are clearly fake.


You can just state that this must be ran in a development environment. Also, please move this to the end of the README.

Done — simplified to "must be run in a development environment" and moved the section to the end of the README.

- Give syntheticTargetTotal an explicit int type and reduce the seed to ~2,000 rows (a couple thousand) for faster local seeding. - Normalize the incoming now to UTC before deriving any dates so the generated series is timezone-independent. - Simplify the README seeding guard to "must be run in a development environment", move the seeding section to the end of the README, and stop claiming idempotency (reruns add more data since IDs derive from wall-clock time). Co-authored-by: Cursor <cursoragent@cursor.com>

cursor

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want higher recall? High effort reviews run extra passes and find more bugs. A team admin can switch effort levels in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit c4d8523. Configure here.}

Re-running the seed on a different day produces new snowflake IDs but the same deterministic (steam_id, marketplace_slug) pairs, which collide with the partial unique index idx_reversals_steam_id_marketplace_slug. Target that natural key (with the WHERE deleted_at IS NULL predicate) in the ON CONFLICT DO NOTHING clause so reruns skip existing rows instead of raising a unique-constraint error, and correct the README/doc comments to describe the idempotent behavior. Also clamp the synthetic createdAt to models.Epoch before the unsigned snowflake timestamp subtraction so an out-of-range clock can't underflow into a garbage ID.

ZukwiZ · 2026-07-13T09:51:05Z

hey @zedimytch, nothing new needed on this one since the last round — think it's good to go whenever you have a sec 🙏

zedimytch

Thanks for this — the overall design is solid. Builds, go vet is clean, and both internal/devseed tests pass against local Postgres. Requesting changes for three issues:

Seeding breaks the local CSFloat ingestor's cursor. csfloatIngestor.sync resumes from MAX(reporter_internal_id) over csfloat rows. ~80% of synthetic rows are csfloat with reporter IDs ~2,900,001–2,902,000, so after seeding, a locally running ingestor resumes from ~2.9M and silently skips real warnings below that. Note that just setting ReporterInternalID = nil isn't safe either: Postgres sorts NULLs first on DESC, so the cursor would become nil and the ingestor would full-resync every 30 minutes. Suggest using a non-real slug for synthetic rows, or nil reporter IDs plus NULLS LAST in the ingestor query.
The Steam IDs aren't "clearly fake". 76561198000000000 is account ID ~39.7M — real accounts created around 2009. Real IDs don't "sit higher up the range" as the description says. With #71 showing avatars/names/profile links, real people would show up as flagged. A base near the top of the 32-bit account space (e.g. 76561197960265728 + 0xF0000000) still passes SteamID.IsValid and isn't allocated.
Reruns never refresh the data. The (steam_id, marketplace_slug) pairs don't depend on now, so a rerun inserts 0 rows (the rerun test asserts exactly this). A week after seeding, the 7d window is empty and re-running doesn't help. Suggest a -reset flag that deletes rows in the synthetic Steam ID range before inserting (straightforward once #2 gives them a dedicated range), and documenting it in the README.

Minor items are inline. The PR description is also stale: it still says ~9.8k rows and ON CONFLICT (id).

zedimytch · 2026-10-09T18:43:06Z

+			sfTs := createdAtMs - models.Epoch
+			sf := models.Snowflake((sfTs << 22) | uint64(seq))
+
+			reporter := syntheticBaseReporter + uint(steamOffset)


These reporter IDs end up on csfloat rows, and the CSFloat ingestor resumes from MAX(reporter_internal_id) for that slug (ingestors/csfloat/csfloat.go sync). After seeding, a local ingestor would resume from ~2.9M and skip real warnings below it. nil isn't a safe alternative on its own, because Postgres puts NULLs first on ORDER BY ... DESC, which would reset the cursor on every sync.

Fixed in ab4746f. Synthetic rows now use dev- slugs (dev-csfloat, etc.) and have no reporter ID at all. The ingestor only looks at csfloat rows, so seeding can't move its cursor anymore, and I didn't need to change the ingestor.

zedimytch · 2026-10-09T18:43:06Z

+	syntheticRNGSeed      int64  = 42
+	syntheticDays                = 180
+	syntheticTargetTotal  int    = 2000
+	syntheticBaseSteamID  uint64 = 76561198000000000


This works out to account ID ~39.7M, which are real 2009-era accounts, so these IDs aren't actually synthetic. Consider basing off the top of the 32-bit account space, e.g. 76561197960265728 + 0xF0000000, which passes SteamID.IsValid and isn't allocated. A dedicated range would also make a -reset cleanup easy.

Good catch, fixed in ab4746f. The base is now 76561197960265728 + 0xF0000000, as you suggested. All generated IDs (including related IDs) stay inside that range, and a new test checks they're valid and in range.

zedimytch · 2026-10-09T18:43:06Z

+
+// GenerateSynthetic returns a ~6-month dataset (~2,000 rows, at least one
+// per day, gentle sinusoid with occasional spikes / quiet days). Snowflake
+// IDs are unique within the slice and won't collide with real CSV-seeded


nit: "CSV-seeded IDs" doesn't exist in this repo. Looks like a leftover from an earlier iteration.

Removed, thanks. That was left over from an earlier version.

zedimytch · 2026-10-09T18:43:06Z

+		for i := 0; i < counts[d]; i++ {
+			reversedAt := dayStart.Add(time.Duration(rng.Float64() * float64(24*time.Hour)))
+			if uint64(reversedAt.UnixMilli()) > nowMs {
+				reversedAt = now.Add(-1 * time.Minute)


For today's bucket, every timestamp later than now gets clamped to now - 1m here (and createdAt to now below). If you seed early in the day, most of today's rows land on the same minute, which shows up as a spike in the 24h KPI and the recent table. Sampling uniformly in [dayStart, now) for the last day would avoid that.

Fixed in ab4746f. For today, rows are now spread randomly between midnight and now instead of being pushed to one minute. If a report time would land in the future, it gets picked randomly between the reversal time and now. I added a test that seeds at 06:00 and checks today's rows aren't all on the same minute.

zedimytch · 2026-10-09T18:43:06Z

+		if end > len(reversals) {
+			end = len(reversals)
+		}
+		res := db.Clauses(onConflict).Create(reversals[i:end])


Two small things:

ON CONFLICT (steam_id, marketplace_slug) only skips that index. A primary-key collision on id would still fail the whole batch. It's unlikely here, but worth a comment.

The factory opens the DB with gorm logger.Info, so this prints the full SQL for each 1,000-row batch. Consider db.Session(&gorm.Session{Logger: logger.Default.LogMode(logger.Warn)}) for the seed.

Both done:

Added a comment on InsertReversals saying ON CONFLICT only covers the (steam_id, marketplace_slug) index, so an id clash would still fail the batch (ab4746f).

The seed now runs on a Warn-level gorm session, so batches don't print their SQL anymore (94cad1c).

zedimytch · 2026-10-09T18:43:06Z

+		t.Errorf("rerun inserted %d rows, want 0 (idempotent)", n2)
+	}
+
+	if first[0].ID == second[0].ID {


nit: this assertion doesn't test anything about the seeder (different now → different IDs is a given). It can be dropped.

Agreed, dropped it.

zedimytch · 2026-10-09T18:43:07Z

+
+- Must be run in a development environment.
+- Generates a 6-month dataset so the dashboard at `/` has enough data to exercise every period (7d / 30d / 3m / 6m / 1y).
+- Uses a synthetic Steam ID prefix (`76561198000000000`) so generated IDs are clearly fake.


76561198000000000 falls in the range of real accounts (~2009), so "clearly fake" isn't accurate. See the comment on syntheticBaseSteamID.

Updated in 9952335. The README now lists the new unallocated ID range and the dev- slugs instead of saying "clearly fake".

zedimytch · 2026-10-09T18:43:07Z

+- Must be run in a development environment.
+- Generates a 6-month dataset so the dashboard at `/` has enough data to exercise every period (7d / 30d / 3m / 6m / 1y).
+- Uses a synthetic Steam ID prefix (`76561198000000000`) so generated IDs are clearly fake.
+- Is safe to re-run: rows are de-duplicated on `(steam_id, marketplace_slug)` via `ON CONFLICT DO NOTHING`, so a rerun skips rows that already exist instead of raising a unique-constraint error.


"Safe to re-run" is true, but a rerun never adds fresh data because the dedupe key doesn't depend on the date. Developers will likely re-run expecting a refresh. Worth documenting how to reset (ideally via a -reset flag).

Added a -reset flag (94cad1c) and documented it in the README (9952335). It deletes every row in the synthetic Steam ID range, then seeds fresh data ending today. Real rows aren't touched. A plain rerun now prints a hint to use -reset when it skips rows.

zedimytch · 2026-10-09T18:43:07Z

+	logging.Initialize()
+	cfg := config.Load()
+
+	if cfg.Environment != constants.EnvironmentDevelopment {


Environment defaults to development in config.go, so this guard only catches an explicit production setting. Fine for a dev tool, but a check that Database.Host is local (localhost / 127.0.0.1 / docker service name) would make it much harder to seed the wrong database by accident.

Added in 94cad1c. The seed now also refuses to run unless Database.Host is localhost, a loopback IP, a unix socket path, or a one-word docker service name like postgres. Anything with a dot or colon is rejected, which also covers Cloud SQL connection names like project:region:instance. There's a small table test for this.

zedimytch · 2026-10-09T18:43:07Z

+	reversals := devseed.GenerateSynthetic(time.Now().UTC())
+	fmt.Printf("generated %d synthetic reversals (deterministic seed)\n", len(reversals))
+
+	inserted, err := devseed.InsertReversals(f.PublicDB(), reversals)


nit: os.Exit(1) on failure below skips the deferred f.Close(). Harmless at exit, but returning an error from a run() func and exiting in main would keep it consistent.

Done in 94cad1c. The logic now lives in run(), which returns errors, and main is the only place that calls os.Exit. The deferred f.Close() now always runs.

- Use Steam IDs from the unallocated top of the 32-bit account space instead of a range that overlaps real 2009-era accounts - Use dev- prefixed marketplace slugs and no reporter_internal_id so seeding can't move the CSFloat ingestor's resume cursor - Add ResetSynthetic to delete rows in the synthetic Steam ID range - Sample today's rows across [dayStart, now) instead of clamping them to one minute - Document that ON CONFLICT only covers the steam_id/slug index - Drop stale 'CSV-seeded' and 'Finding N' references; remove the no-op snowflake assertion; add tests for reset, isolation, and today's spread Co-authored-by: Morten Byskov <ZukwiZ@users.noreply.github.com>

- -reset deletes existing synthetic rows before inserting so a rerun refreshes the data to end at today - Refuse to seed unless Database.Host is local (loopback, unix socket, or a single-label docker service name) - Run inserts on a Warn-level gorm session so batches don't print SQL - Move logic into run() so deferred cleanup runs on failure Co-authored-by: Morten Byskov <ZukwiZ@users.noreply.github.com>

Co-authored-by: Morten Byskov <ZukwiZ@users.noreply.github.com>

cursor · 2026-10-13T21:06:47Z

@zedimytch thanks for the thorough review! All three main issues are fixed:

Ingestor cursor: synthetic rows now use dev- slugs and no reporter ID, so the CSFloat ingestor never sees them.
Steam IDs: they now come from the unallocated top of the 32-bit account space (76561197960265728 + 0xF0000000 and up), so none are real accounts.
Reruns: a new -reset flag deletes the synthetic rows and reseeds fresh data ending today. It's documented in the README.

I replied to each inline comment too, and updated the PR description (no more ~9.8k rows or ON CONFLICT (id)).

Tested with go test ./... against local Postgres 16. I also ran the seed for real: the first run inserted 2006 rows, a rerun inserted 0, and -reset deleted 2006 and reinserted 2006. A non-local DATABASE_HOST is refused.

Commits: ab4746f, 94cad1c, 9952335.

cursor Bot reviewed May 27, 2026

View reviewed changes

Comment thread cmd/seed/main.go

zedimytch self-requested a review June 3, 2026 16:54

zedimytch requested changes Jun 22, 2026

View reviewed changes

cursor Bot reviewed Jun 24, 2026

View reviewed changes

Comment thread internal/devseed/synthetic.go Outdated

Comment thread internal/devseed/synthetic.go Outdated

ZukwiZ requested a review from zedimytch June 24, 2026 13:40

zedimytch requested changes Oct 9, 2026

View reviewed changes

cursoragent and others added 3 commits October 13, 2026 21:01

README: document seed reset and synthetic ID range

9952335

Co-authored-by: Morten Byskov <ZukwiZ@users.noreply.github.com>

Conversation

ZukwiZ commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Safety properties

Other details

Test plan

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ZukwiZ commented Jul 13, 2026

Uh oh!

zedimytch left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor Bot Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor Bot Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor Bot Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor Bot Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor Bot Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor Bot Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor Bot Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor Bot Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor Bot Oct 13, 2026

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

cursor Bot Oct 13, 2026

ZukwiZ commented May 27, 2026 •

edited

Loading