DOT NOT MERGE -- RDS/Aurora logical refresh component #262
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add a new component that automates DBLab full refresh using temporary RDS/Aurora database clones created from snapshots. This enables a hassle-free data sync workflow that doesn't impact production. Features: - Create temporary RDS/Aurora clones from latest automated snapshots - Wait for clone availability with proper timeout handling - Trigger DBLab full refresh via API - Poll refresh status until completion - Clean up temporary clones automatically Deployment options: - AWS Lambda with SAM template and EventBridge scheduling - Standalone CLI binary for cron/manual execution - Docker container for containerized environments Includes comprehensive documentation with IAM policy examples and example configuration files.
Add a fully standalone version of the RDS/Aurora refresh tool that can be built and deployed immediately without requiring a DBLab Engine release. This version has no dependencies on DBLab internal packages. The standalone component includes: - Self-contained Go module with minimal dependencies - Makefile for easy building (CLI, Lambda, Docker) - SAM template for AWS Lambda deployment - Dockerfile for container builds - Comprehensive documentation Can be used immediately by: 1. go build -o rds-refresh . 2. ./rds-refresh -config config.yaml
- Add UpdateSourceConfig method to update DBLab source connection before triggering refresh, ensuring DBLab knows the clone endpoint - Add source database credentials (dbName, username, password) to config for DBLab connection configuration - Remove Lambda-specific code, focusing on container/CLI deployment which better suits the long-running nature of refresh operations - Remove duplicate engine-integrated code (engine/internal/rdsrefresh and engine/cmd/rds-refresh) since standalone component is preferred - Update documentation with container deployment options (Docker, ECS Task, Kubernetes CronJob, CLI with cron) - Update workflow to include config update step before refresh trigger
DBLab expects a flat JSON structure matching ConfigProjection fields,
not a nested structure. The API automatically reloads the config after
update (calls reloadFn internally), so no SIGHUP is needed.
Changed from nested:
{"retrieval": {"dbSource": {"host": "..."}}}
To flat projection format:
{"host": "...", "port": 5432, "dbname": "...", ...}
Contributor
Author
|
@cursor review |
NikolayS
changed the title
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
This PR is being reviewed by Cursor Bugbot
Details
You are on the Bugbot Free tier. On this plan, Bugbot will review limited PRs each billing cycle.
To receive Bugbot reviews on all of your PRs, visit the Cursor dashboard to activate Pro and start your 14-day free trial.
The function could return success prematurely if DBLab's status still showed StatusFinished from a previous refresh. Since /full-refresh triggers the operation in a goroutine, polling immediately after could see stale status. Now tracks whether refresh has actually started before accepting StatusFinished as success. This prevents premature clone deletion.
Contributor
Author
|
@cursor review again |
|
Cursor Agent can help with this pull request. Just |
…tates - Add immediate status check before ticker loop in WaitForRefreshComplete to avoid unnecessary delay of up to pollInterval (30s) on first check - Align IsRefreshInProgress with WaitForRefreshComplete by including StatusPending and StatusRenewed as in-progress states, preventing potential duplicate refresh triggers
Contributor
Author
|
@cursor check it again |
|
I've reviewed the Review summary✅ Bug 1: Race condition (premature completion on stale status) - FIXEDThe ✅ Bug 2: Inconsistent refresh status check - FIXEDThe // IsRefreshInProgress checks if a refresh is currently in progress.
// Considers all active states: refreshing, snapshotting, pending, renewed.
func (c *DBLabClient) IsRefreshInProgress(ctx context.Context) (bool, error) {
status, err := c.GetStatus(ctx)
if err != nil {
return false, err
}
switch status.Retrieving.Status {
case StatusRefreshing, StatusSnapshotting, StatusPending, StatusRenewed:
return true, nil
default:
return false, nil
}
}✅ Bug 3: Delayed first status check - FIXEDThe // immediate first check
if done, err := checkStatus(); err != nil {
return err
} else if done {
return nil
}
ticker := time.NewTicker(pollInterval)No further issues found. The code looks ready for review from a correctness standpoint. Learn more about Cursor Agents |
Contributor
Author
|
@agneum could you please have a look at this (of course, we need to bring this to gitlab and test thoroughly) |
- Add problem/solution section with ASCII diagram - Add prerequisites section - Restructure quick start with numbered steps - Add configuration reference table - Simplify scheduling examples (Docker cron, K8s, ECS) - Add network requirements diagram - Add cost estimates - Improve troubleshooting section
Contributor
Author
|
@cursor review the instructions in README, and scrutinize it -- any potential issues? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Note
Introduces a standalone
rds-refreshtool that clones RDS/Aurora from snapshots, updates DBLab source, triggers full refresh, and cleans up—plus Docker/CLI, configs, and docs.rds-refreshmain.go,refresher.go).dblab.go).rds.go).config.go,config.example.yaml).Dockerfile,Makefile, module files (go.mod,go.sum).README.md).engine/configs/rds-refresh.example.yaml.engine/go.mod/go.sumwith AWS SDK (v2) indirect dependencies.Written by Cursor Bugbot for commit 0104395. Configure here.