chore: Add singular data source support to codespec generation #3970

oarbusi · 2025-12-09T14:55:56Z

Description

Adds support for codespec generation to the use it to support singular data source generation

Link to any related issue(s): CLOUDP-363703

Type of change:

Bug fix (non-breaking change which fixes an issue). Please, add the "bug" label to the PR.
New feature (non-breaking change which adds functionality). Please, add the "enhancement" label to the PR. A migration guide must be created or updated if the new feature will go in a major version.
Breaking change (fix or feature that would cause existing functionality to not work as expected). Please, add the "breaking change" label to the PR. A migration guide must be created or updated.
This change requires a documentation update
Documentation fix/enhancement

Required Checklist:

I have signed the MongoDB CLA
I have read the contributing guides
I have checked that this change does not generate any credentials and that they are NOT accidentally logged anywhere.
I have added tests that prove my fix is effective or that my feature works per HashiCorp requirements
I have added any necessary documentation (if appropriate)
I have run make fmt and formatted my code
If changes include deprecations or removals I have added appropriate changelog entries.
If changes include removal or addition of 3rd party GitHub actions, I updated our internal document. Reach out to the APIx Integration slack channel to get access to the internal document.

Further comments

- Add DataSources struct to codespec model - Add DataSources config to config_model - Add datasources section to config.yml for alert_configuration_api - Update api_to_provider_spec_mapper for data source mapping - Update config transformations for data sources - Generate alert_configuration_api model with data source support

Copilot

Pull request overview

This PR adds data source generation support to the codespec system, enabling independent schema configuration for singular and plural data sources alongside existing resource definitions.

Key Changes:

Adds DataSources configuration block to support independent data source schema options (aliases, overrides) separate from resource schemas
Updates APIOperations to make Create and Read operations optional via pointers, supporting data-source-only API resources
Implements data source attribute merging logic that properly handles path parameters and response attributes with alias support

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tools/codegen/models/org_service_account_secret_api.yaml	Removes unused `update: null` operation declaration
tools/codegen/models/alert_configuration_api.yaml	Adds comprehensive data source schema definition with read/list operations and attribute specifications
tools/codegen/config/config_model.go	Introduces `DataSources` struct to config model for independent data source generation
tools/codegen/config.yml	Adds data source configuration block with read/list operations and alias mapping for alert configurations
tools/codegen/codespec/model.go	Defines `DataSources` and `DataSourceSchema` structs; updates `APIOperations` to use pointers for optional operations
tools/codegen/gofilegen/resource/resource_file.go	Updates operation model conversion to handle pointer-based operations
tools/codegen/gofilegen/resource/resource_file_test.go	Updates test fixtures to use pointer-based `APIOperation` structs
tools/codegen/codespec/config.go	Implements data source transformation logic and refactors path parameter aliasing to support both resources and data sources
tools/codegen/codespec/config_test.go	Updates test fixtures to use pointer-based `APIOperation` structs
tools/codegen/codespec/api_to_provider_spec_mapper.go	Implements `apiSpecToDataSourcesModel` function and attribute merging logic for data source generation
tools/codegen/codespec/api_to_provider_spec_mapper_test.go	Updates test fixtures to use pointer-based `APIOperation` structs

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

tools/codegen/codespec/config.go

tools/codegen/codespec/api_to_provider_spec_mapper.go

AgustinBettati

Thanks for splitting this PR, helped a lot for review.
Overall LGTM, leaving some suggestions and questions for improvements.

AgustinBettati · 2025-12-10T09:09:36Z

tools/codegen/codespec/api_to_provider_spec_mapper_test.go


 	simpleTestResourceOperations = codespec.APIOperations{
-		Create: codespec.APIOperation{
+		Create: &codespec.APIOperation{


would add a unit test in which a config defines data sources configuration and we can assert the resulting model

tools/codegen/codespec/api_to_provider_spec_mapper.go

AgustinBettati · 2025-12-10T09:21:30Z

tools/codegen/codespec/api_to_provider_spec_mapper.go

+
+	// Add path params as required (they identify the data source)
+	// Apply aliases to path params during merge
+	for i := range pathParams {


q: For all merging logic, can we reuse existing addOrUpdate logic we use for resource model generation in merge_attributes.go?
We could also move data source merge function into this file.

I think moving data source merge functions to merge_attributes.go makes sense, but I would keep separate merge functions, because data source logic is much simpler, and the addOrUpdate function is designed for resource-specific complexity (POST/PATCH/response merging) that doesn't apply to data sources

moved in ca348a7

AgustinBettati · 2025-12-10T09:24:05Z

tools/codegen/codespec/api_to_provider_spec_mapper.go

+// Path params are enforced as required; response attributes are marked computed.
+// Aliases are applied to both path params and response attributes during merge to properly detect duplicates.
+// If duplicates exist (same TFSchemaName after aliasing), Required always wins over Computed.
+func mergeDataSourceAttributes(pathParams, responseAttrs Attributes, aliases map[string]string) Attributes {


We do have a step during defined in dataSourceTransformations which calls aliasTransformation, I like this as it reuses all logic with resource model generation. Is there a reason why this function is also handling aliasing? Would strongly favour handling alias in a single place after all merging of attributes is done (similar to resource model generation).

I did this differently to be able to handle the case where we want to alias a path param and a response attribute that are the same but have different names in the path and in the response. For example:

schema: aliases: alertConfigId: integrationId id: integrationId

The problem with aliasing AFTER merge:

Path param: alertConfigId → TFSchemaName: alert_config_id (Required)

Response attr: id → TFSchemaName: id (Computed)

After merge: both alert_config_id and id exist

aliasTransformation changes alert_config_id → integration_id and id → integration_id
Result: TWO attributes with TFSchemaName = integration_id ❌

Why aliasing DURING merge works:

Path param: alertConfigId → alias to integration_id (Required)

Response attr: id → integration_id (Computed)

During merge: duplicate detected → Required wins

Result: ONE integration_id attribute (Required) ✅

The aliasTransformation doesn't handle duplicate detection, it just renames. So for data sources that we want to alias like in the example above, we need to handle it differently.

Let me know your thoughts, i wouldn't block this PR for this, so if we come up with a different solution, we can do it in a follow-up PR

Is this more advanced aliasing feature only supported for data sources? My main doubts comes from diff between resource and data source generation, and hence the difference it what aliasing they support.
I would keep both consistent, we can create a CLOUDP to support this more advanced alias in both implementations and prioritise as we see the need.

This case only happens in the data source because we have the same field as a path param(required input) and as a response attribute(computed output), and there we want to align the name of both fields while avoiding duplicates. In the case of the resource, we only have it in the response attribute, so a computed field, and there's no possibility of duplicates. Hope it makes sense why we need it for the data source only

AgustinBettati · 2025-12-10T09:31:41Z

tools/codegen/codespec/api_to_provider_spec_mapper.go

+	return result
+}
+
+func sortAttributesRecursive(attrs *Attributes) {


Any reason why we need this dedicated sorting step? For reference for resource generation we dont have a dedicated sorting step, buildResourceAttrs does a simple sorting when iterating over schema properties.

refactored this in ca348a7

tools/codegen/codespec/config.go

AgustinBettati · 2025-12-10T09:38:30Z

tools/codegen/codespec/api_to_provider_spec_mapper.go

+	// Process List operation if defined
+	if dsConfig.List != nil {
+		// Extract list operation for description
+		if oasListOp, err := extractOp(spec.Paths, dsConfig.List); err == nil && oasListOp != nil {
+			pluralDescription = &oasListOp.Description
+		}
+
+		listOp = &APIOperation{
+			HTTPMethod: dsConfig.List.Method,
+			Path:       dsConfig.List.Path, // alias will be applied later by transformations helper
+		}
+	}


So right now we only process path params of the read, but not of the list right? Would make clear in PR description what we support related to singular data source model and what is currently pending for the plural (knowing path params of list operation + query params). cc @maastha

This is to implement codespec generation for singular data source only. Changed the title to say singular data source explicitly

AgustinBettati · 2025-12-10T10:08:48Z

tools/codegen/models/alert_configuration_api.yaml

+                    attributes:
+                        - string: {}
+                          description: Uniform Resource Locator (URL) that points another API resource to which this response has some relationship. This URL often begins with `https://cloud.mongodb.com/api/atlas`.
+                          computed_optional_required: optional


spotted in other PR (#3971 (comment)), nested attributes are being marked as Optional/Required in the data source model

Nice catch! working on it

fixed in f7f90de

maastha · 2025-12-10T17:20:49Z

tools/codegen/models/org_service_account_secret_api.yaml

    read:
        http_method: GET
        path: /api/atlas/v2/orgs/{orgId}/serviceAccounts/{clientId}
-    update: null


can you also update the model for org service account so we know it works as expected?

maastha · 2025-12-10T17:27:08Z

tools/codegen/codespec/config_test.go

-				Create: codespec.APIOperation{},
-				Read:   codespec.APIOperation{},
+				Create: &codespec.APIOperation{},
+				Read:   &codespec.APIOperation{},


can you add a test here for aliasing as well for resource & data source?

github-actions bot added the enhancement label Dec 9, 2025

oarbusi changed the title ~~feat: Add data source support to codespec generation~~ chore: Add data source support to codespec generation Dec 9, 2025

github-actions bot removed the enhancement label Dec 9, 2025

oarbusi mentioned this pull request Dec 9, 2025

chore: Add code generation for data sources #3971

Open

13 tasks

fix pointer issue

daa3f35

oarbusi marked this pull request as ready for review December 9, 2025 15:32

oarbusi requested a review from a team as a code owner December 9, 2025 15:32

Copilot AI review requested due to automatic review settings December 9, 2025 15:32

Copilot AI reviewed Dec 9, 2025

View reviewed changes

oarbusi mentioned this pull request Dec 10, 2025

chore: Adds singular datasource auto-generation (WIP) #3956

Closed

13 tasks

AgustinBettati approved these changes Dec 10, 2025

View reviewed changes

AgustinBettati reviewed Dec 10, 2025

View reviewed changes

oarbusi changed the title ~~chore: Add data source support to codespec generation~~ chore: Add singular data source support to codespec generation Dec 10, 2025

oarbusi added 3 commits December 10, 2025 12:23

set all nested attributes as computed

f7f90de

refactor merge_attributes

ca348a7

remove TODO without CLOUDP

cb34e3f

maastha reviewed Dec 10, 2025

View reviewed changes

chore: Add singular data source support to codespec generation #3970

Are you sure you want to change the base?

chore: Add singular data source support to codespec generation #3970

Conversation

oarbusi commented Dec 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change:

Required Checklist:

Further comments

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

AgustinBettati left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

oarbusi commented Dec 9, 2025 •

edited

Loading