NAV
shell

Introduction

This is project that Nebo15 uses to manage it’s infrastructure.

We use:

Why DigitalOcean? Because it’s cheap, friendly and reliable.

Infrastructure overview

Repository and Project Naming

Everybody must name new projects in a agreed way, so we can make assumptions about project, server purpose and deployed environment based on a single string.

Generally repository name should look something like this: {project_name}[.{subproject_name}].{project_type} => {repo_name}, where:

There are 5 known repo types:

In our GitHub account only forked repos can be named in a different way. For Composer-included repos we replace dots (.) with dashes (-).

mbank.api
mbank.web
mbank.layout
mbank.emails.layout
mbank.persona.api
mbank.webclient.web
mbank.ios
mbank.android

Environments

For all projects we have 3 environments:

Branch Naming

All our environments are mapped into git branches. It helps us to keep in mind where code will be provisioned whenever we add a new commit. And helps to get rid of messy branch->environment mapping.

Server Naming

Servers are named based on their hostname and thus a DNS host name. This helps us to avoid PTR records issues.

Initially we name them in the following way: {server_id}.nebo15.com, where:

DNS Records Naming

When we are adding a new server to our pool, we automatically adding a new DNS record to our nebo15.com domain. If you need to map another subdomain or domain to the server, you can simply setup a CNAME DNS record and make corresponding changes in nginx settings.

gandalf.forza.md CNAME gandalf.nebo15.com

Generally we have a convention on naming end-domain endpoints:

Single Responsibility

For production environment we are sticking to “single VM responsibility” strategy. It means that we do not deploy more than one project to a single DO droplet. It helps to scale better and simplifies maintenance of our projects.

For development environment we can have multiple projects on one VM, but still we are trying to avoid it. VM’s are cheaper than cost of developer work.

SSH Authorization

All SSH access should be done trough middleware Arkenstone Auth server. It means that to connect, for example, to example.nebo15.com, you need to run:

Arkenstone will automatically:

Security

Never commit passwords, private keys, API keys or anything like to a git repos. If you did something like this, than you must change all credentials that was affected.

Continuous Integration

Testing

All our applications have minimum code coverage requirement (75%). On each commit we will trigger a build and test on Travis-CI server, that will run following checks:

If one of checks is failed, than you should not merge your feature branch into repo. (And, actually, you won’t be able to.)

Delivery

Non-production environment trigger deployment each time code in corresponding branch is changed. It means that sandbox and beta environment will be always up to date with latest source code.

Initialization

Arkenstone will create a DigitalOcean droplet with specified parameters, using last DO Ubuntu LTS image. For production environment backups is turned on. Only key that is added to server is a Arkenstone master key. Also it should be assigned to a new floating IP, so we would be able to hotswap server in case of critical accidents.

After that it will set a CloudConfig to this VM, that will init Puppet base configuration on this server:

Deploying a Project

  1. Notify Slack that deployment is started.
  2. Asks you to authorize our Arkenstone GitHub app.
  3. Generate a SSH key pair for GitHub repo on a application-server.
  4. Add this key to a GitHub Deploy Keys.
  5. Setup a virtual host to access GitHub that uses this keys from this server (in ~/.ssh/config).
  6. Fetch project git repository to a /www/{project_name}/releases/latest folder.
  7. Change branch to a deployment environment.
  8. Make a project copy (without .git folder) to a /www/{project_name}/releases/{commit_id}-{deploy_timestamp}/ and cd to it.
  9. Provision users ssh keys and passwords from a Arkenstone master-servers.
  10. Fetch all Puppet dependencies and apply project config.
  11. Apply a Puppet config for this project.
  12. Fetch all Puppet dependencies.
  13. Run /www/{project_name}/releases/{commit_id}-{deploy_timestamp}/bin/build.sh.
  14. Create symlink from /www/{project_name}/releases/{commit_id}-{deploy_timestamp}/ to a /www/{project_name}/releases/current.
  15. Notify NewRelic, GitHub and Slack that deployment is finished.
  16. Add a project server to it’s load balancer.

All deploy logs are stored in /var/logs/www/{project_name}/releases/{commit_id}-{deploy_timestamp}/deploy.log.

Deploying an update

Deployment for a changes are very familiar. Just re-run steps 3-10 from previous section.

Service Discovery and Load Balancing

Each time you deploy a application we will try to find a appropriate load balancer for it and add a new node to it’s config. It helps to scale projects really fast.

Users

All Arkenstone users and their changes are automatically provisioned to a main SSH authorization server and to all application servers that user have access to.

List all active Users

GET /users
{
  users: [
    {name: "andrew", public_keys: ["ssh-rsa lsslkjsjlsldl"], password_hash: "sdsdsdsdsd", email:"email@examile.com"}
  ]
}

Create a User

User name is unique string. We are stripping email from public key, to avoid Linux key conflicts.

POST /users
{
  name: "andrew",
  public_keys: ["ssh-rsa lsslkjsjlsldl"],
  password_hash: "sdsdsdsdsd",
  email:"email@examile.com"
}

Get a User

GET /users/:name
{
  name: "andrew",
  public_keys: ["ssh-rsa lsslkjsjlsldl"],
  password_hash: "sdsdsdsdsd",
  email:"email@examile.com"
}

Update a User

PUT /users/:name
{
  name: "andrew",
  public_keys: ["ssh-rsa lsslkjsjlsldl"],
  password_hash: "sdsdsdsdsd",
  email:"email@examile.com"
}

Delete a User

DELETE /users/:name

Application Servers

List all Application Servers

GET /servers
{
  deployments: [
    {
      repo: ":repo_name",
      environment: ":environment",
      deployments: [
        {commit_id: "", time: "", user: ":user_id", is_current: true}
      ]
    }
  ],
  ssh: {
    users: [:user_id, :user_id, :user_id]
  },
  droplet: {
    region: ":do_region",
    size: ":do_size",
    ipv6: true,
    private_networking: true,
    ip: "",
    name: "",
    private_ip: ""
  }
}

Create and provision a new Application Server

POST /servers
{
  deployments: [
    {
      repo: ":repo_name",
      environment: ":environment"
    }
  ],
  ssh: {
    users: [:user_id, :user_id, :user_id]
  },
  droplet: {
    region: ":do_region",
    size: ":do_size",
    ipv6: true,
    private_networking: true
  }
}

Provision changes to a Application Server

PUT /servers/:id
{
  deployments: [
    {
      repo: ":repo_name",
      environment: ":environment"
    }
  ],
  ssh: {
    users: [
      :user_id
    ]
  }
}

Delete an Application Server

DELETE /servers/:id

Halt an Application Server

HALT /servers/:id

Deployments

List all Deployments for a Server

GET /servers/:id/deployments
{
  deployments: [
    {
      repo: ":repo_name",
      environment: ":environment",
      deployments: [
        {commit_id: "", time: "", user: ":user_id", is_current: true}
      ]
    }
  ]
}

Get Project deployments list for an Application Server

GET /servers/:id/deployments/:repo_name
{
  repo: ":repo_name",
  environment: ":environment",
  deployments: [
    {commit_id: "", time: "", user: ":user_id", is_current: true}
  ]
}

Get Project deployment data for an Application Server

GET /servers/:id/deployments/:repo_name/:commit_id
{
  commit_id: "",
  time: "",
  user: ":user_id",
  log: "",
  is_current: true
}

Deploy new version of Application

POST GET /servers/:id/deployments/:repo_name
{
  commit_id: ":commit_id",
  is_current: true
}

Rollback to a previous version of Application

Also this allows to deploy specific version of application to a server.

PUT /servers/:id/deployments/:repo_name/:commit_id
{
  is_current: true
}

Get interactive Deployment log

GET /servers/:id/deployments/:repo_name/:commit_id/log

Webhooks

POST /webhooks/github/ POST /webhooks/slack/ POST /webhooks/newrelic/

Arkenstone Client API

Deploy a project

List all Deployments

GET /deployments
{
  deployments: [
    {
      repo: ":repo_name",
      environment: ":environment",
      deployments: [
        {commit_id: "", time: "", user: ":user_id", is_current: true}
      ]
    }
  ]
}

Get Project deployment data for an Application Server

GET /deployments/:repo_name/:commit_id
{
  commit_id: "",
  time: "",
  user: ":user_id",
  log: "",
  is_current: true
}

Deploy new version of Application

POST GET /deployments/:repo_name
{
  commit_id: ":commit_id",
  is_current: true
}

Rollback to a previous version of Application

Also this allows to deploy specific version of application to a server.

PUT /deployments/:repo_name/:commit_id
{
  is_current: true
}

Get interactive Deployment log

GET /deployments/:repo_name/:commit_id/log

Provision user changes and deploy a new project

PUT /settings
{
  deployments: [
    {
      repo: ":repo_name",
      environment: ":environment"
    }
  ],
  ssh: {
    users: [
      :user_id
    ]
  }
}

TODOs

Refs:

shell