Skip to content

Commit 750e51e

Browse files
sawantpritamsawantpritam
committed
e2e tests CDAP-19055 - Override service account details
1 parent fdbf9fa commit 750e51e

28 files changed

Lines changed: 149 additions & 19 deletions

‎pom.xml‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1005,6 +1005,17 @@
10051005
<forkCount>2</forkCount> <!--Number of JVM processes -->
10061006
<reuseForks>true</reuseForks>
10071007
<!--End configuration to run TestRunners in parallel-->
1008+
<environmentVariables>
1009+
<SERVICE_ACCOUNT_TYPE>
1010+
${SERVICE_ACCOUNT_TYPE}
1011+
</SERVICE_ACCOUNT_TYPE>
1012+
<SERVICE_ACCOUNT_FILE_PATH>
1013+
${SERVICE_ACCOUNT_FILE_PATH}
1014+
</SERVICE_ACCOUNT_FILE_PATH>
1015+
<SERVICE_ACCOUNT_JSON>
1016+
${SERVICE_ACCOUNT_JSON}
1017+
</SERVICE_ACCOUNT_JSON>
1018+
</environmentVariables>
10081019
</configuration>
10091020
<executions>
10101021
<execution>

‎src/e2e-test/features/bigquery/sink/BigQuerySinkError.feature‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ Feature: BigQuery sink - Validate BigQuery sink plugin error scenarios
2121
Then Enter BigQuery property reference name
2222
Then Enter BigQuery property projectId "projectId"
2323
Then Enter BigQuery property datasetProjectId "projectId"
24+
Then Override Service account details if set in environment variables
2425
Then Enter BigQuery property dataset "dataset"
2526
Then Enter BigQuery sink property table name
2627
Then Enter BigQuery sink property GCS upload request chunk size "bqInvalidChunkSize"
@@ -32,6 +33,7 @@ Feature: BigQuery sink - Validate BigQuery sink plugin error scenarios
3233
When Sink is BigQuery
3334
Then Open BigQuery sink properties
3435
Then Enter BigQuery property reference name
36+
Then Override Service account details if set in environment variables
3537
Then Enter BigQuery property dataset "bqInvalidSinkDataset"
3638
Then Enter BigQuery sink property table name
3739
Then Verify the BigQuery validation error message for invalid property "dataset"
@@ -42,6 +44,7 @@ Feature: BigQuery sink - Validate BigQuery sink plugin error scenarios
4244
When Sink is BigQuery
4345
Then Open BigQuery sink properties
4446
Then Enter BigQuery property reference name
47+
Then Override Service account details if set in environment variables
4548
Then Enter BigQuery property dataset "dataset"
4649
Then Enter BigQuery property table "bqInvalidSinkTable"
4750
Then Verify the BigQuery validation error message for invalid property "table"
@@ -54,6 +57,7 @@ Feature: BigQuery sink - Validate BigQuery sink plugin error scenarios
5457
Then Enter BigQuery property reference name
5558
Then Enter BigQuery property projectId "projectId"
5659
Then Enter BigQuery property datasetProjectId "projectId"
60+
Then Override Service account details if set in environment variables
5761
Then Enter BigQuery property dataset "dataset"
5862
Then Enter BigQuery sink property table name
5963
Then Enter BigQuery property temporary bucket name "bqInvalidTemporaryBucket"

‎src/e2e-test/features/bigquery/sink/GCSToBigQuery.feature‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,13 +7,15 @@ Feature: BigQuery sink - Verification of GCS to BigQuery successful data transfe
77
When Source is GCS
88
When Sink is BigQuery
99
Then Open GCS source properties
10+
Then Override Service account details if set in environment variables
1011
Then Enter the GCS source mandatory properties
1112
Then Validate "GCS" plugin properties
1213
Then Close the GCS properties
1314
Then Open BigQuery sink properties
1415
Then Enter BigQuery property reference name
1516
Then Enter BigQuery property projectId "projectId"
1617
Then Enter BigQuery property datasetProjectId "projectId"
18+
Then Override Service account details if set in environment variables
1719
Then Enter BigQuery property dataset "dataset"
1820
Then Enter BigQuery sink property table name
1921
Then Enter BiqQuery property encryption key name "cmekBQ" if cmek is enabled
@@ -41,6 +43,7 @@ Feature: BigQuery sink - Verification of GCS to BigQuery successful data transfe
4143
When Sink is BigQuery
4244
Then Open GCS source properties
4345
Then Enter GCS property projectId and reference name
46+
Then Override Service account details if set in environment variables
4447
Then Enter GCS source property path "gcsCsvRangeFile"
4548
Then Toggle GCS source property skip header to true
4649
Then Select GCS property format "csv"
@@ -51,6 +54,7 @@ Feature: BigQuery sink - Verification of GCS to BigQuery successful data transfe
5154
Then Enter BigQuery property reference name
5255
Then Enter BigQuery property projectId "projectId"
5356
Then Enter BigQuery property datasetProjectId "projectId"
57+
Then Override Service account details if set in environment variables
5458
Then Enter BigQuery property dataset "dataset"
5559
Then Enter BigQuery sink property table name
5660
Then Toggle BigQuery sink property truncateTable to true
@@ -82,6 +86,7 @@ Feature: BigQuery sink - Verification of GCS to BigQuery successful data transfe
8286
When Sink is BigQuery
8387
Then Open GCS source properties
8488
Then Enter GCS property projectId and reference name
89+
Then Override Service account details if set in environment variables
8590
Then Enter GCS source property path "gcsCsvRangeFile"
8691
Then Toggle GCS source property skip header to true
8792
Then Select GCS property format "csv"
@@ -92,6 +97,7 @@ Feature: BigQuery sink - Verification of GCS to BigQuery successful data transfe
9297
Then Enter BigQuery property reference name
9398
Then Enter BigQuery property projectId "projectId"
9499
Then Enter BigQuery property datasetProjectId "projectId"
100+
Then Override Service account details if set in environment variables
95101
Then Enter BigQuery property dataset "dataset"
96102
Then Enter BigQuery sink property table name
97103
Then Toggle BigQuery sink property truncateTable to true
@@ -118,6 +124,7 @@ Feature: BigQuery sink - Verification of GCS to BigQuery successful data transfe
118124
When Sink is BigQuery
119125
Then Open GCS source properties
120126
Then Enter GCS property projectId and reference name
127+
Then Override Service account details if set in environment variables
121128
Then Enter GCS source property path "gcsCsvRangeFile"
122129
Then Toggle GCS source property skip header to true
123130
Then Select GCS property format "csv"
@@ -128,6 +135,7 @@ Feature: BigQuery sink - Verification of GCS to BigQuery successful data transfe
128135
Then Enter BigQuery property reference name
129136
Then Enter BigQuery property projectId "projectId"
130137
Then Enter BigQuery property datasetProjectId "projectId"
138+
Then Override Service account details if set in environment variables
131139
Then Enter BigQuery property dataset "dataset"
132140
Then Enter BigQuery sink property table name
133141
Then Enter BigQuery sink property GCS upload request chunk size "bqChunkSize"

‎src/e2e-test/features/bigquery/sink/GCSToBigQuery_WithMacro.feature‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -32,14 +32,14 @@ Feature: BigQuery sink - Verification of GCS to BigQuery successful data transfe
3232
Then Save the pipeline
3333
Then Preview and run the pipeline
3434
Then Enter runtime argument value "projectId" for key "gcsProjectId"
35-
Then Enter runtime argument value "serviceAccountAutoDetect" for key "gcsServiceAccount"
35+
Then Enter runtime argument value "serviceAccountFilePath" for key "gcsServiceAccount"
3636
Then Enter runtime argument value "gcsCsvFile" for GCS source property path key "gcsSourcePath"
3737
Then Enter runtime argument value "gcsSkipHeaderTrue" for key "gcsSkipHeader"
3838
Then Enter runtime argument value "csvFormat" for key "gcsFormat"
3939
Then Enter runtime argument value "gcsCSVFileOutputSchema" for key "gcsOutputSchema"
4040
Then Enter runtime argument value "projectId" for key "bqProjectId"
4141
Then Enter runtime argument value "projectId" for key "bqDatasetProjectId"
42-
Then Enter runtime argument value "serviceAccountAutoDetect" for key "bqServiceAccount"
42+
Then Enter runtime argument value "serviceAccountFilePath" for key "bqServiceAccount"
4343
Then Enter runtime argument value "dataset" for key "bqDataset"
4444
Then Enter runtime argument value for BigQuery sink table name key "bqTargetTable"
4545
Then Enter runtime argument value "cmekBQ" for BigQuery cmek property key "cmekBQ" if BQ cmek is enabled
@@ -51,14 +51,14 @@ Feature: BigQuery sink - Verification of GCS to BigQuery successful data transfe
5151
Then Deploy the pipeline
5252
Then Run the Pipeline in Runtime
5353
Then Enter runtime argument value "projectId" for key "gcsProjectId"
54-
Then Enter runtime argument value "serviceAccountAutoDetect" for key "gcsServiceAccount"
54+
Then Enter runtime argument value "serviceAccountFilePath" for key "gcsServiceAccount"
5555
Then Enter runtime argument value "gcsCsvFile" for GCS source property path key "gcsSourcePath"
5656
Then Enter runtime argument value "gcsSkipHeaderTrue" for key "gcsSkipHeader"
5757
Then Enter runtime argument value "csvFormat" for key "gcsFormat"
5858
Then Enter runtime argument value "gcsCSVFileOutputSchema" for key "gcsOutputSchema"
5959
Then Enter runtime argument value "projectId" for key "bqProjectId"
6060
Then Enter runtime argument value "projectId" for key "bqDatasetProjectId"
61-
Then Enter runtime argument value "serviceAccountAutoDetect" for key "bqServiceAccount"
61+
Then Enter runtime argument value "serviceAccountFilePath" for key "bqServiceAccount"
6262
Then Enter runtime argument value "dataset" for key "bqDataset"
6363
Then Enter runtime argument value for BigQuery sink table name key "bqTargetTable"
6464
Then Enter runtime argument value "cmekBQ" for BigQuery cmek property key "cmekBQ" if BQ cmek is enabled

‎src/e2e-test/features/bigquery/source/BigQuerySourceError.feature‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ Feature: BigQuery source - Validate BigQuery source plugin error scenarios
1919
When Source is BigQuery
2020
Then Open BigQuery source properties
2121
Then Enter BigQuery property reference name
22+
Then Override Service account details if set in environment variables
2223
Then Enter the BigQuery source properties with incorrect property "<property>" value "<value>"
2324
Then Validate BigQuery source incorrect property error for table "<property>" value "<value>"
2425
Examples:
@@ -50,6 +51,7 @@ Feature: BigQuery source - Validate BigQuery source plugin error scenarios
5051
Then Enter BigQuery property reference name
5152
Then Enter BigQuery property projectId "projectId"
5253
Then Enter BigQuery property datasetProjectId "projectId"
54+
Then Override Service account details if set in environment variables
5355
Then Enter BigQuery property dataset "dataset"
5456
Then Enter BigQuery source property table name
5557
Then Enter BigQuery property temporary bucket name "bqInvalidTemporaryBucket"

‎src/e2e-test/features/bigquery/source/BigQueryToBigQuery.feature‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,13 +10,15 @@ Feature: BigQuery source - Verification of BigQuery to BigQuery successful data
1010
Then Enter BigQuery property reference name
1111
Then Enter BigQuery property projectId "projectId"
1212
Then Enter BigQuery property datasetProjectId "projectId"
13+
Then Override Service account details if set in environment variables
1314
Then Enter BigQuery property dataset "dataset"
1415
Then Enter BigQuery source property table name
1516
Then Enter BigQuery source property filter "bqFilter"
1617
Then Validate output schema with expectedSchema "bqSourceSchema"
1718
Then Validate "BigQuery" plugin properties
1819
Then Close the BigQuery properties
1920
Then Open BigQuery sink properties
21+
Then Override Service account details if set in environment variables
2022
Then Enter the BigQuery sink mandatory properties
2123
Then Validate "BigQuery" plugin properties
2224
Then Close the BigQuery properties
@@ -44,13 +46,15 @@ Feature: BigQuery source - Verification of BigQuery to BigQuery successful data
4446
Then Enter BigQuery property reference name
4547
Then Enter BigQuery property projectId "projectId"
4648
Then Enter BigQuery property datasetProjectId "projectId"
49+
Then Override Service account details if set in environment variables
4750
Then Enter BigQuery property dataset "dataset"
4851
Then Enter BigQuery source property table name
4952
Then Enter BigQuery source property filter "bqInvalidFilter"
5053
Then Validate output schema with expectedSchema "bqSourceSchema"
5154
Then Validate "BigQuery" plugin properties
5255
Then Close the BigQuery properties
5356
Then Open BigQuery sink properties
57+
Then Override Service account details if set in environment variables
5458
Then Enter the BigQuery sink mandatory properties
5559
Then Validate "BigQuery" plugin properties
5660
Then Close the BigQuery properties
@@ -68,13 +72,15 @@ Feature: BigQuery source - Verification of BigQuery to BigQuery successful data
6872
Then Enter BigQuery property reference name
6973
Then Enter BigQuery property projectId "projectId"
7074
Then Enter BigQuery property datasetProjectId "projectId"
75+
Then Override Service account details if set in environment variables
7176
Then Enter BigQuery property dataset "dataset"
7277
Then Enter BigQuery source property table name
7378
Then Enter BigQuery source properties partitionStartDate and partitionEndDate
7479
Then Validate output schema with expectedSchema "bqPartitionSourceSchema"
7580
Then Validate "BigQuery" plugin properties
7681
Then Close the BigQuery properties
7782
Then Open BigQuery sink properties
83+
Then Override Service account details if set in environment variables
7884
Then Enter the BigQuery sink mandatory properties
7985
Then Validate "BigQuery" plugin properties
8086
Then Close the BigQuery properties
@@ -102,13 +108,15 @@ Feature: BigQuery source - Verification of BigQuery to BigQuery successful data
102108
Then Enter BigQuery property reference name
103109
Then Enter BigQuery property projectId "projectId"
104110
Then Enter BigQuery property datasetProjectId "projectId"
111+
Then Override Service account details if set in environment variables
105112
Then Enter BigQuery property dataset "dataset"
106113
Then Enter BigQuery source property table name
107114
Then Enter BigQuery source properties partitionStartDate "bqFuturePartitionStartDate" and partitionEndDate "bqFuturePartitionEndDate"
108115
Then Validate output schema with expectedSchema "bqPartitionSourceSchema"
109116
Then Validate "BigQuery" plugin properties
110117
Then Close the BigQuery properties
111118
Then Open BigQuery sink properties
119+
Then Override Service account details if set in environment variables
112120
Then Enter the BigQuery sink mandatory properties
113121
Then Validate "BigQuery" plugin properties
114122
Then Close the BigQuery properties
@@ -130,6 +138,7 @@ Feature: BigQuery source - Verification of BigQuery to BigQuery successful data
130138
Then Enter BigQuery property reference name
131139
Then Enter BigQuery property projectId "projectId"
132140
Then Enter BigQuery property datasetProjectId "projectId"
141+
Then Override Service account details if set in environment variables
133142
Then Enter BigQuery property dataset "dataset"
134143
Then Enter BigQuery source property table name
135144
Then Validate output schema with expectedSchema "bqSourceSchemaDatatype"
@@ -139,6 +148,7 @@ Feature: BigQuery source - Verification of BigQuery to BigQuery successful data
139148
Then Enter BigQuery property reference name
140149
Then Enter BigQuery property projectId "projectId"
141150
Then Enter BigQuery property datasetProjectId "projectId"
151+
Then Override Service account details if set in environment variables
142152
Then Enter BigQuery property dataset "dataset"
143153
Then Enter BigQuery sink property table name
144154
Then Toggle BigQuery sink property truncateTable to true
@@ -167,6 +177,7 @@ Feature: BigQuery source - Verification of BigQuery to BigQuery successful data
167177
Then Enter BigQuery property reference name
168178
Then Enter BigQuery property projectId "projectId"
169179
Then Enter BigQuery property datasetProjectId "projectId"
180+
Then Override Service account details if set in environment variables
170181
Then Enter BigQuery property dataset "dataset"
171182
Then Enter BigQuery source property table name
172183
Then Validate output schema with expectedSchema "bqSourceSchemaDatatype"
@@ -176,6 +187,7 @@ Feature: BigQuery source - Verification of BigQuery to BigQuery successful data
176187
Then Enter BigQuery property reference name
177188
Then Enter BigQuery property projectId "projectId"
178189
Then Enter BigQuery property datasetProjectId "projectId"
190+
Then Override Service account details if set in environment variables
179191
Then Enter BigQuery property dataset "dataset"
180192
Then Enter BigQuery sink property table name
181193
Then Toggle BigQuery sink property truncateTable to true

‎src/e2e-test/features/bigquery/source/BigQueryToGCS.feature‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,11 +12,13 @@ Feature: BigQuery source - Verification of BigQuery to GCS successful data trans
1212
Then Enter BigQuery property datasetProjectId "projectId"
1313
Then Enter BigQuery property dataset "dataset"
1414
Then Enter BigQuery source property table name
15+
Then Override Service account details if set in environment variables
1516
Then Enter BiqQuery property encryption key name "cmekBQ" if cmek is enabled
1617
Then Validate output schema with expectedSchema "bqSourceSchema"
1718
Then Validate "BigQuery" plugin properties
1819
Then Close the BigQuery properties
1920
Then Open GCS sink properties
21+
Then Override Service account details if set in environment variables
2022
Then Enter the GCS sink mandatory properties
2123
Then Validate "GCS" plugin properties
2224
Then Close the GCS properties
@@ -43,12 +45,14 @@ Feature: BigQuery source - Verification of BigQuery to GCS successful data trans
4345
Then Enter BigQuery property reference name
4446
Then Enter BigQuery property projectId "projectId"
4547
Then Enter BigQuery property datasetProjectId "projectId"
48+
Then Override Service account details if set in environment variables
4649
Then Enter BigQuery property dataset "dataset"
4750
Then Enter BigQuery source property table name
4851
Then Validate output schema with expectedSchema "bqSourceSchemaDatatype"
4952
Then Validate "BigQuery" plugin properties
5053
Then Close the BigQuery properties
5154
Then Open GCS sink properties
55+
Then Override Service account details if set in environment variables
5256
Then Enter the GCS sink mandatory properties
5357
Then Validate "GCS" plugin properties
5458
Then Close the GCS properties
@@ -75,6 +79,7 @@ Feature: BigQuery source - Verification of BigQuery to GCS successful data trans
7579
Then Enter BigQuery property reference name
7680
Then Enter BigQuery property projectId "projectId"
7781
Then Enter BigQuery property datasetProjectId "projectId"
82+
Then Override Service account details if set in environment variables
7883
Then Enter BigQuery property dataset "dataset"
7984
Then Enter BigQuery source property table name as view
8085
Then Toggle BigQuery source property enable querying views to true
@@ -84,6 +89,7 @@ Feature: BigQuery source - Verification of BigQuery to GCS successful data trans
8489
Then Validate "BigQuery" plugin properties
8590
Then Close the BigQuery properties
8691
Then Open GCS sink properties
92+
Then Override Service account details if set in environment variables
8793
Then Enter the GCS sink mandatory properties
8894
Then Validate "GCS" plugin properties
8995
Then Close the GCS properties

0 commit comments

Comments
 (0)