Money Forward Developers Blog

Automatically Creating Parameters on AWS Parameter Store & Refreshing Parameters without Restarting.

I. Background

A cloud-native application, unlike a monolithic application, comprises independent services across virtual machines, containers, and regions. Managing configurations for numerous interdependent services is challenging. Centralized configuration is crucial for distributed cloud-native applications.

On the Send Cloud Invoice (SCI) project, we decided to use AWS Parameter Store as a centralized configuration service because we can take some advantages.

Although AWS Parameter Store provides many benefits, we also faced some struggles when using it for manually creating parameters, parameters hot reloading, etc.

The followings are solutions to solve these problems.

II. Automatically Creating Parameters on the AWS Parameter Store

1. Problems

  • We need to manually create parameters on the AWS Parameter Store in many environments (beta, staging, production).
  • If we forget to create them, our application cannot start. Each time we release, we need to bookmark all the parameters that need to be created.

2. Solution

  • Idea:
    • In each environment, we have configuration files (app_config_{env}) to store parameters.
    • When triggering CI/CD(Circle CI), Lambda will read these configuration files and create new parameters on the AWS parameter store.
  • Steps
    1. Create new parameters in app_config_{env} along with the application code.
    2. Create a Pull Request to build on the beta/staging/production environment.
    3. Trigger Circle CI run CI/CD.
    4. Circle CI calls the Lambda function which reads the configuration file (app_config_{env}) and creates new parameters in the AWS Parameter Store.
    5. Build and push the docker image to the registry.
    6. Deploy new code by running an ECS task.
  • Note:
    • This solution cannot be applied to sensitive data (username, password of database, etc).
    • This solution only supports creating new parameters on the AWS Parameter Store, not updating them although we want to update the values via AWS UI instead of updating the configuration files.

III. Refreshing Parameters without Restarting the Service

1. Problems

  • Currently, each time we want to change the values of the parameters, we have to restart our application to make the parameters take effect on our application.
  • Automation testing team sometimes needs to change the values of the parameters for testing without restarting our application.
  • For operation, we want to investigate the bugs by changing the log level to debug.

2. Solution

  • Our application will connect to the AWS Parameter Store to get all the parameters.
  • If we change the values of the parameters, the AWS Parameter Store will send an event to a Redis Stream. Our application listens to this Redis Stream and refreshes.

Steps:

  1. When a user updates the AWS Parameter Store, it will fire a new event to Event Bridge.
  2. After receiving the event, Event Bridge will trigger a Lambda function.
  3. This Lambda function will push a message to a topic in the Redis Stream.
  4. Service instances that use the Spring Cloud library will listen to the topic. Then, it calls AWS Parameter Store to get the latest parameters and refreshes Bean which uses modified parameters.

3. Implementation

  • We use Redis Stream as our pub/sub model because it is a lightweight and simple solution for low-volume data streams:
    • Redis Streams is a built-in data structure in Redis, and also a popular in-memory data store.
    • Redis Streams is a simple, lightweight, low-latency solution for handling real-time data streams.
    • Redis Streams is best suited for use cases where data volume is relatively low and the processing requirements are not too complex.
    • Redis Streams provides a unified storage and processing model, allowing users to store and process data within the same system.
    • Redis Streams supports consumer groups, which allow multiple consumers to read from the same stream in a coordinated way.
  • Library
  <dependency>
    <groupId>io.awspring.cloud</groupId>
    <artifactId>spring-cloud-starter-aws-parameter-store-config</artifactId>
    <version>2.4.4</version>
  </dependency>
  • For example, we need to add annotation @RefreshScope to Bean object in Spring framework:
    • Refresh Beans With @ConfigurationProperties
  • Refresh Beans With @Value 

IV. Documents:

Published-date