Settle

Docs

By August 20, 2018 No Comments

Purpose of this document is to help new developers onboard into Settle Continuous Deployment Pipeline.

What are Settle Apps ?

Settle Apps are an easy way for you as the developers to create Web-Facing and Internal, production ready applications in form of Web UI or REST API, that can dynamically scale and facilitate traffic they face without you having to worry about anything beside commiting the code into the repository.

What do I need to know ?

  • Git, how to push code into the repository / specific branch, how to make pull request.
  • In case of REST API applications – C# or JavaScript basics
  • In case of Web UI – JavaScript, React, Redux or Typescript, and C# basics

What kind of application can I exactly create ?

Examples:

  • Exchange Ticker Tracker and History Data Collection
  • Trading bots
  • Market makers
  • Signal providing
  • Social Media query and analyzer
  • Chatbots, social media bots
  • Calculation, small data aggregation networks and engines
  • AI / ML on datasets generated by before mentioned examples

How do I start ?

First decide what kind of application you would want to create, then in what form you want to expose its functionality.

You can choose one or more for a single project:

  • Web UI
  • Public REST API
  • Private REST API

Public API’s are accessible from internet.

Private API’s can be only accessed via the public API’s, but can’t be reached from internet. (They have access to internet, even though are not accessible from it)

Next select language you want to code your application in, currently you can pick either:

  • C# .NET Core 2.1

After that contact / create group conversation with Asmodat#8166 and roni#6621 on discord https://discord.gg/vt4ak7Z after which you will be given access to the private bitbucket repository and jenkins build server credentials.

You will need to provide your email address and create associated with it Bitbucket account https://bitbucket.org beforehand.

Lastly you will need to provide your working hours in UTC which will be used to make build agent accessible to you in those hours (Its used to create resources for your application in cloud)

Deployment Flowchart

Bitbucket Repository

Once you receive access you will be able to access your repository via this address: https://bitbucket.org/concourseqapps/

Your repository will be already preset with demo example and ready to be deployed.

You will work with two branches

  • master – used for production code
  • develop – used for testing code

You will have full access to the develop branch, but you will not be able to access master unless you create a pull request

We recommend to create SSH key in order to access your repository, you can add it in user settings of your account: https://bitbucket.org/account/user/<YOUR_ACCOUNT_NAME>/ssh-keys/

To create a key you can use PuTTY Key  Generator: https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html

To create key you simply click “Generate” button then you can paste the public key into the Bitbucket SSH settings.

To generate private key that will be used by git to push your code into the repository please select Conversion → Export OpenSSH Key

To clone the repository you can use following commands

ssh-agent bash -c ‘ssh-add <OPEN_SSH_KEY_LOCATION>; git clone [email protected]:concourseqapps/<REPOSITORY_NAME>.git’
cd <REPOSITORY_NAME>/

To add, commit and push all changes you can run following command

git add -A && git commit -m “Here you can type any commit message you want!”
ssh-agent bash -c ‘ssh-add <OPEN_SSH_KEY_LOCATION>; git push origin develop’

You don’t have to bother with console commands if you use GUI client, we recommend Git Tortoise for windows: https://tortoisegit.org/

You can deploy applications pushed to the develop branch to test via Jenkins that they work properly, break them or even destroy after you finished as they are not used in the production.

When your code is stable and ready to be presented to wider audience you you have to create pull request from develop to master branch.

You can only merge your code to the master branch if you receive at least 2 pull request approvals, to receive them you have to perform following actions:

  • Run the Jenkins Test Pipeline – after successful build your pull request will be automatically approved
  • Ask one of the admins (Roni / Asmodat) to review the code.

Jenkins Automation Server

Jenkins is a software that will allow you to turn your code into real living application, you can access it via following link: https://www.production.concourseq.net/jenkins/ (you will be provided credentials)

Once you login you will see following control panel

You will have Log Read access to 2 pipelines, that build, test and deploy code published to the master branch.

  • <APP_NAMESPACE>
  • <APP_NAMESPACE>-Destroy

You will have Log Read and Build Execute access to 2 pipelines, that build, test and deploy code published to the develop branch.

  • <APP_NAMESPACE>-Test
  • <APP_NAMESPACE>-Test-Destroy

To build your application you click “Build Now” button after clicking the link to the pipeline in the control panel and to read the log you will click on the build number in the “Build History” tab then on the “Console Output”.

Console Output will contain an error message describing what went wrong in case build fails and you will be able to use it in order to debug any issues preventing your app from being deployed.

Build Flow

Build flow is an automated process performed by jenkins node (after user initiates that action by clicking “Build Now” button in the jenkins console) and consist of following steps:

Build:

  • Clone Repository from develop or master branch
  • Check integrity of the files in scripts directory, ensure they were not modified
  • Execute deploy.sh script from the scripts directory

Deploy.sh Script:

  • Build source code of all applications
  • Publish the code into the output directories (Artifacts)
  • Execute Unit Tests
  • Create Docker Images and upload them to private docker repository
  • Deploy Resources of each app based on uploaded docker images and ensure applications are accessible (This process is parallelized)
  • Execute Integration tests

Persistent Storage – S3

Applications don’t have persistent storage of their own and are expected to fail under high network stress or resource overutilization conditions then reboot without any previous state (settle apps are auto scaled and load balanced).

To persist any kind of state / files you should use S3 bucket which will be assigned to each of your application under certain rules.

By default all files are encrypted and tight permissions applied so you will have to use only specific folders.

You will receive personal KMS Key ID (It has form of a GUID), that your application can to use to encrypt sensitive / secret data within S3 Bucket.

Thanks to this even if something happens and bucket permissions malfunction for any reason no one will be able to decrypt data belonging to your applications.

Folder permission structure

 

Settle apps have Read/Write/Delete permissions to all folders they can access (each project can have more then one API/UI application):

  • app-cluster/<APP_UID>/public/* – accessible by all public and private applications in your project
  • app-cluster/<APP_UID>/private/* – accessible by all private applications in your project
  • app-cluster/<APP_UID>/<APP_NAME>/* – accessible only by particular applications in your project

Secrets

You might want to provision your applications with secrets, keys or api codes, but you don’t want anyone to see them in your repository (even if its private). Simplest way to solve that issue is to encrypt them and commit along the code and for that purpose we provide a tool simplifying that process for you.

Latest release of the CDHelper tool from github: https://github.com/asmodat/CDHelper/releases

It’s a self contained application for Windows, Mac and Linux, you will not need any external libraries, and the code is fully open-source, so you can verify what it does and even build yourself.

This tool allows you to provision encrypted secrets into your repository. Even though repository you will be given is private and only readable to you and our administrators you don’t want things like private keys or API keys be stored within config files as plaintext.

If you are using windows machine simply download the release zip file, extract anywhere you want and add the path to executable file called CDHelper.exe into your PATH environment variables.

If you are using linux machine you can download and install this tool by running following commands:

sudo -s

cd /usr/local/src

rm -fv ./CDHelper-linux-x64.zip

wget https://github.com/asmodat/CDHelper/releases/download/0.0.4/CDHelper-linux-x64.zip

rm -rfv /usr/local/bin/CDHelper

unzip CDHelper-linux-x64.zip -d /usr/local/bin/CDHelper

chmod -R 777 /usr/local/bin/CDHelper

nano /etc/profile.d/execute.sh

#save at the end of the file then click Ctrl+O, [enter] Ctrl+X

export PATH=/usr/local/bin/CDHelper:$PATH

 

Now add yet another environmental variable key called ‘Settle<APP_UID>KeyPath’ then assign it variable that is a path (including the filename, that should end with .aess extension) where your key will be stored. For example: S:\secrets\settle-app-<APP_UID>-secret.aess

Now you can create your own secret encryption/decryption key by running following command using bash or git-bash.

CDHelper aes create-key –path=”$Settle<APP_UID>KeyPath”

Content of the file will have following json format: {“Key”:”XXXXXXXXXXXXXXXXXXXXXXXXX”,”IV”:”XXXXX”,”Padding”:”PKCS7″,”Mode”:”CBC”}

Currently secret decryption key provisioning is a manual process, but it will change in the future into an automated routine, you will have to send the secret you generated to Asmodat#8166 on Discord and and it will be used to provision the jenkins node that will build and deploy your application.

To encrypt your secrets and provision your repository with them go to your cloned Project Repository and in the root of it next to the .gitignore file you will find a file called: secrets.config.json

That file is a JSON array of object with two fields: Source and Destination.

  • Source is an exact path on your PC where you (locally) store you secret file
  • Destination is an relative path within repository where your encrypted secret will be saved

Now you can open git-bash console in the root in the root of your project and execute following command.

CDHelper aes encrypt –config=”./secrets.config.json” –key=”$Settle<APP_UID>KeyPath”

This will encrypt all files specified as Source in the json array and save into the Destination location you specified. Files should be named exactly the same as your original files, but end with the .aes extension.

During the build process jenkins node will use your secret key and to un-encrypt all files in your repository that have .aes extension.

Project Structure

After cloning your project repository, you will see following root folder structure:

  • scripts – folder containing deployment logic specific to your application (you can’t modify it without approval and jenkins will not execute your build if you modify its content)
  • Sources – folder containing your applications – one or many UI or API applications
  • .gitignore – file describes which files should not be committed into the repository, you can learn more bout it here: https://git-scm.com/docs/gitignore
  • secrets.config.json – configuration file used to help you provision your repository with encrypted secrets

Sources contain folders with your applications code and have short unique names aka <APP_NAME>’s which should not be changed. Within except the code you will find two files used in deployment process, which you should not modify:

  • docker-compose.app.yml – describes docker file localisation, environment variables and ports your application needs to have access to communicate with outside world
  • dockerfile – describes how to package and start your application

You can learn more about those here: https://docs.docker.com/engine/reference/builder/ but in the future they will be moved into a scripts directory.

When you receive a repository it will be already provisioned with code samples for each of your UI or API application, all you have to do is modify the code, only thing to remember is that you should never remove api/test/ping endpoints, as they are used for detecting if your application is alive, so if they are removed then service executing your application code will be marked for cleanup / destruction.

Automated Unit and Integration Testing

Currently unit and integration testing is only supported for C# projects we support Unit and Integration testing.

  • Unit Test – tests executed before docker image containing your application is created
  • Integration Tests – tests executed after successful deployment of your application

C# .Net Core 2.0

Demo example contain unit tests managed by NUnit library.

To create a new Test Project Solution you have to place new .NET Core 2.0 Console Application in the <REPOSITORY_NAME>\Sources\Tests folder and that install following nuget packages:

  • NUnit (3.10.1)
  • NUnit3 TestAdapter (3.10.0)
  • Microsoft.NET.Test.Sdk (15.7.2)
  • Asmodat.Standard (1.0.0.9)

After that you can create your new NUnit test class, more about that you can find in here: https://docs.microsoft.com/en-us/dotnet/core/testing/unit-testing-with-nunit#creating-the-first-test

All the above is already prepared for you in a form of demo examples, only thing you have to remember is your test will not be executed unless it contains an IntegrationTest or UnitTest attribute. You don’t have to create your own attributes if you install NuGet package called Asmodat.Standard (1.0.0.9) then you can import the class already containing them located at namespace: AsmodatStandard.Extensions.NUnit.Attributes.

Example: Testing Ping Endpoint of the public API

Vertical Scaling

By default 0.24 vCPU and 0.5GB RAM will be assigned to each of your projects, you can request vertical scaling upgrades up to 4 vCPU (Intel Xeon Cores) and 30GB RAM after producing proof of need to utilize such resources along quality code on github.

Supported Configurations for RAM and CPU per each API or UI Project

CPU Memory Values
0.25 vCPU 0.5GB, 1GB, and 2GB
0.5 vCPU Min. 1GB and Max. 4GB, in 1GB increments
1 vCPU Min. 2GB and Max. 8GB, in 1GB increments
2 vCPU Min. 4GB and Max. 16GB, in 1GB increments
4 vCPU Min. 8GB and Max. 30GB, in 1GB increments

Horizontal Scaling

Default desired task count (Initial number of the same applications created during the deployment) is 1 after providing proof your application needs horizontal scaling and is heavily used this number can be increased.

Currently auto-scaling feature is disabled, but (default) upper limit will be set to 3 tasks per application.

Where is my application ?

After deploying your application by clicking “Build Now” button in the Jenkins management console and successful completion of that process your application will be assigned DNS record via which you will be able to access it:

<APP_NAME> is an application name, it’s the same as the folder name containing the code of your application

<PORT_NUMBER> is a port via which application will be available, currently (by default) Web UI’s are exposed via port 80, Public API’s ia port 8000 and private API’s via port 9000

<APP_UID> is an Unique identifier of your application and has following format

  • In case of test pipeline deployment (code from develop branch): test<NUMBER>
  • In case of production pipeline deployment (code from master branch): prod<NUMBER>

<NUMBER> is a ‘number’ / ID of your application that you will be assigned

Example: http://www.test-3-api-1.public.concourseq.net:8000/api/test/ping

Example: http://www.prod-3-ui-1.public.concourseq.net:80

Where physically is my application ?

Applications are run as cluster of containers on the AWS fargate, you can learn more about it here: https://aws.amazon.com/fargate/

Leave a Reply