MB-61292: Make sure we can read all deks at startup

Before this commit we ignored read key errors, because we
had to support the case when log dir is removed together
with log deks. Now since log deks are stored in config dir,
they can't be removed when logs are removed, so it should
be save to assume that deks must be always readable.

There is another scenario that needs to be kept in mind:
say we have a dek encrypted by aws key, and that aws key
is unavailable at startup, so we can't read that dek.
There are two ways to handle that:
1. Continue to start up, but retry reading deks later;
2. Fail to start up.

Option #1 is hard to implement as the code that uses that dek
should handle the case when dek is not available.

This is another reason why this commit implements option #2.
Note that this scenario was not supported before this commit.

Change-Id: Ib01c009957ae7f413428b38c6f2c32bb19f193db
Reviewed-on: https://review.couchbase.org/c/ns_server/+/221170
Reviewed-by: Navdeep S Boparai <[email protected]>
Well-Formed: Build Bot <[email protected]>
Tested-by: Timofey Barmin <[email protected]>

Author: timofey-barmin

apps/ns_server/src/cb_cluster_secrets.erl

@@ -1642,25 +1642,21 @@ read_all_deks(#state{} = State) ->
              fun (Kind, #{is_enabled := IsEnabled,
                           active_id := ActiveId,
                           dek_ids := DekIds}) ->
-                 {Keys, Errors} = cb_deks:read(Kind, DekIds),
-                 case Errors of
-                    #{ActiveId := Reason} when IsEnabled ->
-                        ?log_warning("Failed to read active ~p DEK: ~p",
-                                     [Kind, Reason]),
-                        %% Not creating that DEK in configuration
-                        %% We can't start using undefined as an active key
-                        %% because this can lead to data being decrypted
-                        %% which is probably not what we want here. We should
-                        %% rather show an error or crash instead of silently
-                        %% decrypting the data.
-                        %% It is also possible that all deks were removed
-                        %% intentinally (with the data). In this case we should
-                        %% not crash but rather ignore these deks at all.
-                        false;
-                    #{} ->
-                        %% Ignoring other key errors and hoping that those keys
-                        %% will not be needed for data decryption.
-                        {true, new_dek_info(ActiveId, Keys, IsEnabled)}
+                 Snapshot = deks_config_snapshot(Kind),
+                 case call_dek_callback(encryption_method_callback, Kind,
+                                        [Snapshot]) of
+                     {succ, {ok, _}} ->
+                         {Keys, Errors} = cb_deks:read(Kind, DekIds),
+                         case maps:size(Errors) > 0 of
+                             true ->
+                                 ?log_error("Failed to read ~p keys: ~p",
+                                            [Kind, Errors]),
+                                 exit({failed_to_read_keys, Errors});
+                             false ->
+                                 {true, new_dek_info(ActiveId, Keys, IsEnabled)}
+                         end;
+                     {succ, {error, not_found}} ->
+                         false
                  end
              end, Term),
     State#state{deks = Deks}.