1  Introduction to ndexr

ndexr builds on the AWS services EC2, IAM, and Route53. To get started, you need an AWS account with billing set up. ndexr integrates these services, providing a comprehensive experience for building and deploying software.

When building an online business, consulting firm, or hosting applications to showcase your skills, you must understand critical concepts to weave everything together. The best place I know how to do this is AWS.

If you wanted to build everything Netflix does for a single customer on one computer, you probably could. You would only have difficulty scaling to the number of computers required to make such a system run at scale. When you are accessing a website, you rarely have any idea what kind of computer resources are supporting the application at that current moment in time. You can build beautifully large applications with extremely minimal resources that are deeply complex from the outside perspective. The only thing preventing you from hosting asdf.io, hello.com, and dev.asdf.io on a single Raspberry Pi in your living room is knowing how to do it.

But we need to start somewhere. As I have been building ndexr, the closest competitor I can see is shinyapps.io, a website created by Posit. However, I like to think of ndexr as something better than shinyapps.io and on the way to competitor status with Posit Connect. We can see what ndexr does and compare it to shinyapps.io. The following are the top features included in the Professional tier at shinyapps.io, which costs $349/month and $3,860/yr.

1.1 shinyapps.io or ndexr

Unlimited Applications and 10,000 Active Hours

According to Ian Pylvainen with Posit Support -

An Active Hour is an hour that an instance of an application is up and serving requests. Whether you have one or one hundred people using one session of your Shiny app for an hour, you are still only using one active hour. However, if you have only one person actively using two different Shiny apps for one hour at the same time, you are consuming two active hours in that time. If you have configured your application for multiple sessions to enhance performance as more people use it (available in Basic plans and above only), each session of the application uses active hours as if it were a separate app.

So here’s the first set of terminology I want to make a dent in. When I say instance, I use it in terms of how someone using AWS EC2 uses it and make a point to distinguish language to make a point in learning. When I say instance, I mean an EC2 Instance. An instance is created when I go through the steps to start my server. And once I pay AWS money for a running server, I say I have an EC2 instance up and running. I am paying 30 cents an hour to run a c6a.2xlarge instance.

This is a point because what he calls an instance I would think of as a Shiny session—the consumption of one port for a single application launch. It is an important step in deploying an application, generally managed by someone other than the developer. It’s where you move from data science and analytics to engineering, the heart of network security.

So from now on, his version of the instance, in this case, I will call session. When we start an R session, we’re consuming a port where we can connect our application to a network somewhere. Maybe it is only so that your web browser can access the application. Then, your browser makes an HTTP request to the application that serves the content you are creating. This is one R session running R. This one R session cannot execute parallel commands without calling an external service like C, Rust, or other R sessions.

A session is one copy of your application running with finite resources. What is the difference between one session and two? Technically, the consumption of a port. When you start your Shiny application in RStudio, if you have not defined explicitly which port your applications run on, Shiny will find a random open port to run on. Which port is consumed and how by which application becomes essential to understand. So, according to the definition, you can have one application running with about 13 concurrent users or two applications - one with five simultaneous users and the other with 8, etc.

Authentication

There are multiple ways of restricting access to an application. The easiest is generally to put a password on it. Passwords can restrict access to an application without hiding the application in more sophisticated ways like by device. This allows your application to be open to the world while remaining secure. shinyapps.io offers this, but the way that I do it is a little bit different. I am building a template so you can go to Amazon Cognito, set up your user base, and then use a pre-configured shiny app with the Cognito code already built in. This is a work in progress. On access, incoming HTTP requests to the server are routed through Nginx, which can be integrated with an active directory, safelist IPs, and password-protect browser access. These features are built-in with open-source software like this, and ndexr helps and builds automation around these workflows.

Account Sharing

Allow multiple shiny developers to use your account, which obviously, if you have partners or employees, would be helpful for each user to access the account for management where acceptable. Ndexr does this by giving you all of the open-source tools needed to do this yourself. Each server is set up with Nginx already configured to your domain, with a shiny application load balanced behind it. The shiny app needs some good modules-in-modules examples, like button groups, table rows, and async examples. But really, the pieces are all there. You are immediately given access to an Ubuntu server with Nginx, docker, and docker-compose set up with Shiny already running. You can break and rebuild from there, given how projects are structured with docker-compose, and Make the parent folder and a services folder for each service in your docker-compose. yaml file.

Performance Boost

shinyapps.io gives you up to 8 GiB RAM performance boost for when your apps need it. But you should check this list of instance prices. Look up a t3.large for general purpose, or if you want to play around with a c6a.48xlarge or p3 instance, have fun. Also, 350/mo is a minimum of 45 cents an hour. Check out how cheap a c6a.2xlarge is for that amount. ;)

Custom Domains

Instead myapp.shinyapps.io you can have ndexr.io or some other domain. This means that DNS management is built in. How ndexr does this is still under construction, but basically, it takes your nginx configuration and matches it up with your DNS record. The way it does this is by 1. Checking if you have a free elastic IP and then creating one for you if you don’t. 2. Updating your Route53 with an A record for the domain/subdomain you picked. This is less than ideal, and soon you will be able to match your Route53 domains with n domains running on a single instance directed to other websites, databases, shiny sessions, etc.

1.2 Launching and managing your server with ndexr

What we are going to do is recreate this entire process but manually with ndexr. Instead of shinyapps.io, it will be your server and fully own your deployment pipeline. This is both interesting and rewarding and also significantly cheaper. ndexr builds a lot of this for you on AWS.

1.2.1 Buy a Domain (or domains - you can have as many as you want on one server) on Route 53:

Where: AWS console

The first thing you will want to do is grab ownership of a domain. A domain is a name for an application space on the internet. A domain will cost money, but you can have as many sub-domains as you like for free. Also, because domains resolve to an IP, you can have a domain set up with an A record like ndexr.io and then link a subdomain to it using a CNAME linking api.ndexr.io to www.ndexr.io but where nginx determines on your server what domain goes to which ports.

1.2.2 Activate Command Line Access in:

Where: AWS console

  • Obtain/Delete Elastic IP in EC2.
  • Open Cost Explorer in Billing.

1.2.3 Configure AWS Secrets for Access Credentials:

Where: Create Credentials in AWS Console and Insert them into the ndexr console.

Go to IAM to create a user and generate credentials for command line usage. The suggested steps follow -

Create a policy for ndexr: IAM > Policies

  1. Click on Create policy.

  2. Click on JSON and attach the policy. This gives ndexr full access to your Route53, EC2, and Cost Explorer consoles in AWS. I’m happy to work with you if anyone wants more restrictive permissions. This is just a place where I haven’t spent exhaustive time because constantly updating a policy gets tedious as I build.

    {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "ec2:*",
                "route53:*",
                "ce:*"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}

  3. Click Next.

  4. Policy name: ndexrRoute53,EC2,andCostExplorerService

  5. Description: This policy allows ndexr to be able to access your AWS console in order to be able to manage your account.

  6. Add tags: ndexr

Create a User

  1. Click on Users
  2. Click on Create user
  3. Create a username ndexr
  4. Click Attach policies directly.
  5. Search Permissions policies for ndexrRoute53,EC2,andCostExplorerService
  6. Check the prior policy; you should only see one.
  7. Click Next
  8. Add new tag: ndexr
  9. Click Create user

Create Credentials for ndexr User

  1. Click on your user created prior in IAM > Users
  2. Click on Security Credentials
  3. Click on Create access key
  4. Click on Third-party service
    • You will see a warning:

      • Alternative recommended

        As a best practice, use temporary security credentials (IAM roles) instead of creating long-term credentials like access keys, and don’t create AWS account root user access keys. Learn more.

      • You can revoke access to ndexr at any time by either deleting the user in the AWS Console or removing the policy we created. Once I have time to look at the alternative authentication methods, I will update the site to follow them.

  5. Set description tag: ndexr
  6. Click on Create access key
  7. Click on Download .csv file
  8. Provide the Access key and Secret access key in the ndexr console
    • Access keys are stored encrypted using AWS Secrets Manager

1.2.4 Create a Key for Your Server:

Where: ndexr console

  1. Click on Key Files
  2. Key Name: ndexr-first-app
  3. Click on Import SSH Keys
  4. Note you can download or destroy the key. If you destroy it, it is gone forever except for use on the servers you use it to access, or where you have saved it. Keyfiles need restricted permissions to be used. chmod 400 ~/ndexr-first-app.pem
  5. Click Dismiss

1.2.5 Security Group / Firewall Settings:

Where: ndexr console

  1. Click on Manage Security Groups

  2. Security Group Name: ndexr-first-app

  3. Click Create

  4. Click Import Security Groups

  5. Click the down arrow for ndexr-first-app

    • Default ports open to the world: 22, 80, 443. This is totally safe with the way things are set up; however, if you want to be even safer, you can do the following to restrict access to your location.
    1. Click Revoke for 22, 443, and 80.
    2. Port: 22
    3. IP: Your IP address (displayed in console) - note you have to append /32 to your IP. So my current IP at Starbucks, according to this rule, is 65.153.22.75/32
    4. Click down arrow
    5. Note you have created access to port 22 only from your current location and nowhere else.
    6. Repeat for ports 80 and 443
    7. Repeat for locations you are comfortable allowing access.

1.2.6 Launch a Server:

Where: ndexr console

  1. Key Name: ndexr-first-app
  2. Security Group: ndexr-first-app
  3. Amazon Machine Image: Use the default.
  4. Instance Storage: 150
  5. Instance Type:
  6. Click Start Server - an ish formula is
    • Note that AWS begins charging you by the second and you will accrue \((\text{Seconds Running}/3600)*\text{Instance Price} + \text{Elastic IP Rate}\)
    • Elastic IP Rate is $0.005 per IP per hour

1.3 Manage Your Server

Where: ndexr console

Set up HTTPS (Required for Stripe and Basic Security)

  1. Click on Manage Servers
  2. Click on Refresh
  3. Select the IP of your new server
  4. Subdomain: www.
  5. Domain: yourroute53domain.com
  6. Click Connect
    • This will begin talking to Route 53, provisioning an Elastic IP, and updating your NGINX configuration on your server with HTTPS.
    • This will be updated in a future version to manage multiple domains and sub-domains on a single server, but you can learn how to do that now by exploring how sites-enabled works in NGINX.
    • Warning: I need to work a little here, because you can try to connect before the server is even on. If this button fails because the server isn’t provisioned yet, do it again. Remember, an Elastic IP is created at this step regardless of whether the certbot succeeds.
  7. When you see “Awesome, that worked!” exit and click on Manage Servers, and then Refresh again -
    • You should see your server with the domain we created

Manage Server State

Use Start, Stop, Reboot, and Terminate to control the server. Stop avoids charges for a running server. Terminate destroys the server and its contents. You can enable Termination Protection in the AWS EC2 Console. I haven’t added a button for it yet. Note you will still accrue an elastic IP charge unless you remove it in the AWS Console.

Notes:

  1. You can only enter the backend through port 22, where SSH runs.
    • Primary configuration file: /etc/ssh/sshd_config
  2. Ports 80 and 443 are managed by nginx on your server, which
    • Primary configuration files:
      • /etc/nginx/nginx.conf/
      • /etc/nginx/sites-enabled/default

Modify Instance Type

Use Modify to change the instance type. This will begin the process of turning off your server, restarting it and then modifying the hardware your server is running on. It truly is like being able to take your laptop and turn it into a supercomputer for a time and then back again. Truly marvelous. Note you should be aware of your instance type costs and stop the server when you don’t need it running.

1.4 Update Configuration Files

One of the things I am doing is building automation around configuration files I commonly use. I think you should familiarize yourself with NGINX and Docker Compose, and here is why - NGINX connects outside users to the applications that Docker Compose manages. You need to understand ports because you have to know how to say - you, this user who is requesting dev.example.com, needs to go to the applications running on ports 9000-9099, and example.com needs to go to one of the apps at 9100-9199. It is reasonable to have one server that handles multiple domains for multiple teams and projects or to segment those teams and projects over various servers.

Ok, so let’s start with make. I like Make, and there is probably a better solution. Some people go as far as to say you’re a bad dev if you’re still using it. I don’t freaking know, honestly, but it works. And here’s why it’s nice.

With ndexr, I automate my development workflow in a shiny app, and on AWS, I am using Make to automate common shell snippets together, which helps automate a larger workflow. It’s software that lets us tell other types of software what commands to run and in which order more quickly than writing shell scripts because we can compose them nicely and neatly in one file. On the next server, Make is already installed. You can cd into cd ~/public and run make up to start the default application at ~/public/services/console which is the skeleton project of console.ndexr.io - and I will be adding to it constantly.

This Makefile will always be replaceable with the Makefile ~/public/Makefile on an ndexr server. I’ll try to keep the docs ahead of the server itself. But here’s the connection

Running make up profile=ndexrio scale=10 in consumes all ports between 9000-9009, because Make tells which ports Docker Compose should consume and nginx directs users to all active ports in the port range it is told to monitor. I mean this because on the nginx side you can “block out” a range of ports - in our case by default

Makefile

# Docker Compose Makefile

# Variables
profile := ndexrio
scale := 2

# Targets
.PHONY: up down

up:
    @echo "Building Docker Compose services..."
    docker compose --profile=$(profile) build
    @echo "Starting Docker Compose services in the background..."
    docker compose --profile=$(profile) up -d --remove-orphans
    @echo "Scaling the specified service to $(scale) instances..."
    docker compose --profile=$(profile) up --scale $(profile)=$(scale) -d

down:
    @echo "Stopping and removing Docker Compose services..."
    docker compose --profile=$(profile) down --remove-orphans

NGINX

Click Pull and note the default NGINX configuration. This is good to go for you after you set up your domain. This is where all non-SSH traffic goes. Note you can manage access to the server in AWS by listing your IP address, or you can deny all and allow in the nginx configuration to manage IP access. Filepath: /etc/nginx/sites-enabled/default

When you push an update the file is uploaded to the server, and nginx is restarted.

upstream shinyapp {
    server localhost:9000;
    server localhost:9001;
    ...
}

server {
    server_name rxedn.com www.rxedn.com;
    large_client_header_buffers 4 32k;

    location / {
        proxy_pass http://shinyapp/;
    ...
    }
}

Docker and Docker Compose

version: '3.9'

services:
  ndexrio:
    restart: unless-stopped
    build:
      context: ./services/console
    ports:
      - 9000-9009:8000
    network_mode: bridge
    command: ["R", "-e", "options(ndexr_site='ndexrio');shiny::runApp('/app/src')"]
    profiles:
      - ndexrio

1.5 Access Applications. (on your server behind your domain)

1.5.1 Log into RStudio:

Keep both RStudio and vscode whitelisted to your IP in the nginx configuration. deny all should be removed only if you understand what it means. You know the URL of your applications because you can see it in your nginx configuration. i.e., example.com/rstudio

  • Default credentials (change immediately) using passwd: RStudio uses system users for authentication by default but can be integrated with other Federated identities and Active Directory.
    • Username: rbox
    • Password: password

1.5.2 Log into Visual Studio Code:

  • Password: your_password
    • Instructions for changing the vscode password need to be written or parameterized.