Merge pull request #3132 from wireapp/release_2023-03-06_18_57

Release 2023-03-06 - (expected chart version 4.34.0)

Author: smatting

.github/workflows/ci.yml

@@ -11,8 +11,8 @@ jobs:
      - uses: actions/checkout@v2
        with:
          submodules: true
-     - uses: cachix/install-nix-action@v14.1
-     - uses: cachix/cachix-action@v10
+     - uses: cachix/install-nix-action@v20
+     - uses: cachix/cachix-action@v12
        with:
          name: wire-server
          signingKey: '${{ secrets.CACHIX_SIGNING_KEY }}'
@@ -32,8 +32,8 @@ jobs:
      - uses: actions/checkout@v2
        with:
          submodules: true
-     - uses: cachix/install-nix-action@v14.1
-     - uses: cachix/cachix-action@v10
+     - uses: cachix/install-nix-action@v20
+     - uses: cachix/cachix-action@v12
        with:
          name: wire-server
          signingKey: '${{ secrets.CACHIX_SIGNING_KEY }}'

.gitignore

@@ -59,9 +59,6 @@ spar.integration-aws.yaml
 integration-aws.yaml
 DOCKER_ID*
 swagger-ui
-!charts/nginz/static/swagger-ui
-
-services/nginz/integration-test/conf/nginz/zwagger-ui/*
 
 deploy/dockerephemeral/build/airdock_base-all/
 deploy/dockerephemeral/build/airdock_base/
@@ -112,3 +109,6 @@ result-*
 
 /integration-ca-key.pem
 /integration-ca.pem
+
+services/nginz/third_party/headers-more-nginx-module
+services/nginz/third_party/nginx-module-vts

CHANGELOG.md

@@ -1,3 +1,122 @@
+# [2023-03-06] (Chart Release 4.34.0)
+
+## Release notes
+
+
+* In (the unlikely) case your server config file contains `setWhitelist:`, you need to change this before the upgrade!  It used to refer to a whitelisting service, which is now replaced with a local list of allowed domains and phone numbers.  See [docs](https://docs.wire.com/developer/reference/user/activation.html?highlight=whitelist#phone-email-whitelist) for details.  Migration path: add new config fields; upgrade, remove old config fields. (#3043)
+
+* The coturn Helm chart has been promoted to *beta* level stability. (#3078)
+
+
+## API changes
+
+
+* API v3 is now supported. The new MLS endpoints introduced in API v3 have been removed, and are now only available under v4. (#3122)
+
+
+## Features
+
+
+* Add internal endpoints of `cargohold`, `galley`, `legalhold` and `spar` to the Swagger docs for internal endpoints. (#3007)
+
+* The coturn container image included in the coturn Helm chart was updated to
+  version `4.6.0-wireapp.4`.
+
+  With this version of coturn, the Prometheus metrics endpoint has been
+  updated, and the `turn_active_allocations` metric label has been *renamed* to
+  `turn_total_allocations`. (#3078)
+
+* Better error message for invalid ID in a credential when uploading MLS key packages (#3102)
+
+* Add Swagger documentation for internal endpoints. It's reachable at the path `/v<n>/api-internal/swagger{-ui,.json}`. (#3003)
+
+* Render one Swagger page per internal endpoint. This superseeds the previous Swagger docs page for all internal endpoints. (#3094)
+
+* Feature flag for Outlook calendar integration (#3025)
+
+* Team feature setting for MLS end-to-end identity was added and server setting `setEnableMls` is exposed via new authorized endpoint `GET /system/settings` (#3082)
+
+
+## Bug fixes and other updates
+
+
+* The container image used for handling online TLS certificate updates in the
+  coturn Helm chart was updated to a version with metadata compatible with
+  containerd. (#3078)
+
+* Fix a bug in the helm chart's nginx-ingress-services / federator Ingress resource introduced in the last release. (#3034)
+
+* Remove overly restricte api check (#3131)
+
+* Typing indicators not working accross federated backends (#3118)
+
+
+## Documentation
+
+
+* Extend the docs on the federation error type (#3045)
+
+* Update SAML/SCIM docs (#3038)
+
+
+## Internal changes
+
+
+* - use exponential backoff for retrying requests to Amazon
+  - also retry in case of server-side rate limiting by Amazon (#3121)
+
+* Also run the 'backoffice' pod in CI (to test it can successfully start) (#3130)
+
+* Make brig-schema a little faster by merging the first 34 schema migrations on fresh installations. (#3099)
+
+* Deflake integration test: metrics (#3053)
+
+* Document in code a function that sends remote Proteus messages (#PR_NOT_FOUND)
+
+* Lower the log level of federator inotify (#3056)
+
+* use Wai's settings for graceful shutdown (#3069)
+
+* CI integration setup time should be reduced: tweak the way cassandra-ephemeral is started (#3052)
+
+* charts: Mark all service/secret/configmap test resources to be re-created by defining them as helm hooks (#3037, #3049)
+
+* New integration test script with support for running end2end tests locally (#3062)
+
+* Bump nixpkgs to latest commit on nixpkgs-unstable branch (#3084)
+
+* Add config to allow to run helm tests for different services in parallel; improve integration test output logs (#3040)
+
+*  Run brig and galley integration tests concurrently (#2825)
+
+* Add wrapper for bitnami/postgresql chart. (#3012)
+
+* Branch on performAction tags for finer-grained CallsFed constraints (#3030)
+
+* Fixed broken stern endpoint `POST i/user/meta-info` (#3035)
+
+* Make stern fail on startup if supported backend api version needs bumping (#3035)
+
+* Automatically track CallsFed constraints via a GHC plugin (#3083)
+
+* Rust library `rusty-jwt-tools` upgraded to latest version (#3112)
+
+* Fixed test of jwt-tools Rust FFI (#3125)
+
+* Enabling warnings for redundant constraints and removing the redundant
+  constraints. (#3009)
+
+* Migrate `/teams/notifications` to use the Servant library. (#3020)
+
+* Split polysemy `Members` constraints into multiple `Member` constraints (#3093)
+
+
+## Federation changes
+
+
+* Use `HsOpenSSL` instead of `tls` for federation communication. (#3051)
+
+
 # [2023-01-26] (Chart Release 4.31.0)
 
 ## Release notes

Makefile

@@ -18,8 +18,9 @@ fake-aws fake-aws-s3 fake-aws-sqs aws-ingress fluent-bit kibana backoffice		\
 calling-test demo-smtp elasticsearch-curator elasticsearch-external				\
 elasticsearch-ephemeral minio-external cassandra-external						\
 nginx-ingress-controller nginx-ingress-services reaper sftd restund coturn		\
-inbucket k8ssandra-test-cluster
+inbucket k8ssandra-test-cluster postgresql
 KIND_CLUSTER_NAME     := wire-server
+HELM_PARALLELISM      ?= 1 # 1 for sequential tests; 6 for all-parallel tests
 
 package ?= all
 EXE_SCHEMA := ./dist/$(package)-schema
@@ -90,6 +91,15 @@ endif
 ci: c db-migrate
 	./hack/bin/cabal-run-integration.sh $(package)
 
+.PHONY: sanitize-pr
+sanitize-pr:
+	./hack/bin/generate-local-nix-packages.sh
+	make formatf-all
+	make hlint-inplace-all
+	make git-add-cassandra-schema
+	@git diff-files --quiet -- || ( echo "There are unstaged changes, please take a look, consider committing them, and try again."; exit 1 )
+	@git diff-index --quiet --cached HEAD -- || ( echo "There are staged changes, please take a look, consider committing them, and try again."; exit 1 )
+
 .PHONY: cabal-fmt
 cabal-fmt:
 	./hack/bin/cabal-fmt.sh $(package)
@@ -235,11 +245,11 @@ db-migrate-package:
 
 # Usage:
 #
-# Reset all keyspaces
-# make db-reset
+# Migrate all keyspaces and reset the ES index
+# make db-migrate
 #
-# Reset keyspace for only one service, say galley:
-# make db-reset package=galley
+# Migrate keyspace for only one service, say galley:
+# make db-migrate package=galley
 .PHONY: db-reset
 db-reset: c
 	@echo "Make sure you have ./deploy/dockerephemeral/run.sh running in another window!"
@@ -248,31 +258,42 @@ ifeq ($(package), all)
 	./dist/galley-schema --keyspace galley_test --replication-factor 1 --reset
 	./dist/gundeck-schema --keyspace gundeck_test --replication-factor 1 --reset
 	./dist/spar-schema --keyspace spar_test --replication-factor 1 --reset
+ifeq ($(INTEGRATION_FEDERATION_TESTS), 1)
+	./dist/brig-schema --keyspace brig_test2 --replication-factor 1 --reset
+	./dist/galley-schema --keyspace galley_test2 --replication-factor 1 --reset
+	./dist/gundeck-schema --keyspace gundeck_test2 --replication-factor 1 --reset
+	./dist/spar-schema --keyspace spar_test2 --replication-factor 1 --reset
+endif
 else
 	$(EXE_SCHEMA) --keyspace $(package)_test --replication-factor 1 --reset
+ifeq ($(INTEGRATION_FEDERATION_TESTS), 1)
+	$(EXE_SCHEMA) --keyspace $(package)_test2 --replication-factor 1 --reset
+endif
 endif
+	./dist/brig-index reset --elasticsearch-index directory_test --elasticsearch-server http://localhost:9200 > /dev/null
+	./dist/brig-index reset --elasticsearch-index directory_test2 --elasticsearch-server http://localhost:9200 > /dev/null
 
 # Usage:
 #
-# Migrate all keyspaces
+# Migrate all keyspaces and reset the ES index
 # make db-migrate
 #
 # Migrate keyspace for only one service, say galley:
 # make db-migrate package=galley
 .PHONY: db-migrate
 db-migrate: c
-ifeq ($(package), all)
-	./dist/brig-schema --keyspace brig_test --replication-factor 1
-	./dist/galley-schema --keyspace galley_test --replication-factor 1
-	./dist/gundeck-schema --keyspace gundeck_test --replication-factor 1
-	./dist/spar-schema --keyspace spar_test --replication-factor 1
-# How this check works: https://stackoverflow.com/a/9802777
-else ifeq ($(package), $(filter $(package),brig galley gundeck spar))
-	$(EXE_SCHEMA) --keyspace $(package)_test --replication-factor 1
-else
-	@echo No schema migrations for $(package)
+	./dist/brig-schema --keyspace brig_test --replication-factor 1 > /dev/null
+	./dist/galley-schema --keyspace galley_test --replication-factor 1 > /dev/null
+	./dist/gundeck-schema --keyspace gundeck_test --replication-factor 1 > /dev/null
+	./dist/spar-schema --keyspace spar_test --replication-factor 1 > /dev/null
+ifeq ($(INTEGRATION_FEDERATION_TESTS), 1)
+	./dist/brig-schema --keyspace brig_test2 --replication-factor 1 > /dev/null
+	./dist/galley-schema --keyspace galley_test2 --replication-factor 1 > /dev/null
+	./dist/gundeck-schema --keyspace gundeck_test2 --replication-factor 1 > /dev/null
+	./dist/spar-schema --keyspace spar_test2 --replication-factor 1 > /dev/null
 endif
-
+	./dist/brig-index reset --elasticsearch-index-prefix directory --elasticsearch-server http://localhost:9200 > /dev/null
+	./dist/brig-index reset --elasticsearch-index-prefix directory2 --elasticsearch-server http://localhost:9200 > /dev/null
 
 #################################
 ## dependencies
@@ -315,15 +336,15 @@ kube-integration:  kube-integration-setup kube-integration-test
 
 .PHONY: kube-integration-setup
 kube-integration-setup: charts-integration
-	export NAMESPACE=$(NAMESPACE); ./hack/bin/integration-setup-federation.sh
+	export NAMESPACE=$(NAMESPACE); export HELM_PARALLELISM=$(HELM_PARALLELISM); ./hack/bin/integration-setup-federation.sh
 
 .PHONY: kube-integration-test
 kube-integration-test:
-	export NAMESPACE=$(NAMESPACE); ./hack/bin/integration-test.sh
+	export NAMESPACE=$(NAMESPACE); export HELM_PARALLELISM=$(HELM_PARALLELISM); ./hack/bin/integration-test.sh
 
 .PHONY: kube-integration-teardown
 kube-integration-teardown:
-	export NAMESPACE=$(NAMESPACE); ./hack/bin/integration-teardown-federation.sh
+	export NAMESPACE=$(NAMESPACE); export HELM_PARALLELISM=$(HELM_PARALLELISM); ./hack/bin/integration-teardown-federation.sh
 
 .PHONY: kube-integration-e2e-telepresence
 kube-integration-e2e-telepresence:

cabal.project

@@ -1,4 +1,4 @@
-with-compiler: ghc-8.10.7
+with-compiler: ghc-9.2.4
 
 packages:
     libs/api-bot/

cassandra-schema.cql

@@ -137,8 +137,13 @@ CREATE TABLE galley_test.team_features (
     mls_allowed_ciphersuites set<int>,
     mls_default_ciphersuite int,
     mls_default_protocol int,
+    mls_e2eid_lock_status int,
+    mls_e2eid_status int,
+    mls_e2eid_ver_exp timestamp,
     mls_protocol_toggle_users set<uuid>,
     mls_status int,
+    outlook_cal_integration_lock_status int,
+    outlook_cal_integration_status int,
     search_visibility_inbound_status int,
     search_visibility_status int,
     self_deleting_messages_lock_status int,
@@ -1083,7 +1088,7 @@ CREATE TABLE brig_test.provider (
 CREATE TABLE brig_test.user_keys (
     key text PRIMARY KEY,
     user uuid
-) WITH bloom_filter_fp_chance = 0.01
+) WITH bloom_filter_fp_chance = 0.1
     AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
     AND comment = ''
     AND compaction = {'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
@@ -1124,7 +1129,7 @@ CREATE TABLE brig_test.invitee_info (
     invitee uuid PRIMARY KEY,
     conv uuid,
     inviter uuid
-) WITH bloom_filter_fp_chance = 0.01
+) WITH bloom_filter_fp_chance = 0.1
     AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
     AND comment = ''
     AND compaction = {'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
@@ -1290,7 +1295,7 @@ CREATE TABLE brig_test.user (
     sso_id text,
     status int,
     team uuid
-) WITH bloom_filter_fp_chance = 0.01
+) WITH bloom_filter_fp_chance = 0.1
     AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
     AND comment = ''
     AND compaction = {'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
@@ -1398,7 +1403,7 @@ CREATE TABLE brig_test.password_reset (
     retries int,
     timeout timestamp,
     user uuid
-) WITH bloom_filter_fp_chance = 0.01
+) WITH bloom_filter_fp_chance = 0.1
     AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
     AND comment = ''
     AND compaction = {'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
@@ -1513,7 +1518,7 @@ CREATE TABLE brig_test.connection (
     status int,
     PRIMARY KEY (left, right)
 ) WITH CLUSTERING ORDER BY (right ASC)
-    AND bloom_filter_fp_chance = 0.01
+    AND bloom_filter_fp_chance = 0.1
     AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
     AND comment = ''
     AND compaction = {'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
@@ -1584,7 +1589,7 @@ CREATE TABLE brig_test.activation_keys (
     key_type ascii,
     retries int,
     user uuid
-) WITH bloom_filter_fp_chance = 0.01
+) WITH bloom_filter_fp_chance = 0.1
     AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
     AND comment = ''
     AND compaction = {'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}

charts/brig/templates/configmap.yaml

@@ -270,8 +270,11 @@ data:
       {{- if .setSftListAllServers }}
       setSftListAllServers: {{ .setSftListAllServers }}
       {{- end }}
-      {{- if .setWhitelist }}
-      setWhitelist: {{ toYaml .setWhitelist | nindent 8 }}
+      {{- if .setAllowlistEmailDomains }}
+      setAllowlistEmailDomains: {{ toYaml .setAllowlistEmailDomains | nindent 8 }}
+      {{- end }}
+      {{- if .setAllowlistPhonePrefixes }}
+      setAllowlistPhonePrefixes: {{ toYaml .setAllowlistPhonePrefixes | nindent 8 }}
       {{- end }}
       {{- if .setFeatureFlags }}
       setFeatureFlags: {{ toYaml .setFeatureFlags | nindent 8 }}

charts/brig/templates/tests/brig-integration.yaml

@@ -2,16 +2,17 @@ apiVersion: v1
 kind: Service
 metadata:
   name: "brig-integration"
+  annotations:
+    "helm.sh/hook": post-install
+    "helm.sh/hook-delete-policy": before-hook-creation
   labels:
     app: brig-integration
     chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
     release: {{ .Release.Name }}
     heritage: {{ .Release.Service }}
 spec:
   type: ClusterIP
-  ports:
-    - port: 9000
-      targetPort: 9000
+  clusterIP: None
   selector:
     app: brig-integration
     release: {{ .Release.Name }}
@@ -21,7 +22,7 @@ kind: Pod
 metadata:
   name: "{{ .Release.Name }}-brig-integration"
   annotations:
-    "helm.sh/hook": test-success
+    "helm.sh/hook": test
   labels:
     app: brig-integration
     release: {{ .Release.Name }}