Testing your Amazon Serverless Application Model (SAM) locally with Neo4j, Lambda and Docker

So this is what I want to achieve. You should be able to do the same by the end of the post:

[ ] Install Docker
[ ] Install NPM
[ ] Create Neo4j Docker container
[ ] Docker Network to allow containers to talk to each other
[ ] Install SAM Local (beta)
[ ] Create template.yaml
- [ ] Python
- [ ] JavaScript
- [ ] Java
[ ] Create test handler
- [ ] Invoke driver
  - [ ] Talk about possible problems and how expensive it is to invoke drives?
- [ ] Run read query
  - [ ] Link to driver pages and different queries and transactions
  - [ ] Why we won’t close off the connection in case our lambda is reused
[ ] Test Lambda function

Install Docker

http://lmgtfy.com/?q=install+docker+on+windows/linux/mac

If you’re on a mac:

cmd+space to bring up Spotlight Search, type terminal and press enter
Install Homebrew

$ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Update Homebrew

$ brew update

Install Docker via Homebrew

$ brew cask install docker

cmd+space to bring up Spotlight Search, type docker and press enter
- When it starts up, it’ll create everything in the following location

$ ls -l /usr/local/bin/docker*
lrwxr-xr-x  1 laexample  domain Users  67 Apr 12 14:14 /usr/local/bin/docker -> /Users/laexample/Library/Group Containers/group.com.docker/bin/docker
lrwxr-xr-x  1 laexample  domain Users  75 Apr 12 14:14 /usr/local/bin/docker-compose -> /Users/laexample/Library/Group Containers/group.com.docker/bin/docker-compose
lrwxr-xr-x  1 laexample  domain Users  90 Apr 12 14:14 /usr/local/bin/docker-credential-osxkeychain -> /Users/laexample/Library/Group Containers/group.com.docker/bin/docker-credential-osxkeychain
lrwxr-xr-x  1 laexample  domain Users  75 Apr 12 14:14 /usr/local/bin/docker-machine -> /Users/laexample/Library/Group Containers/group.com.docker/bin/docker-machine

Validate version

$ docker version

[x] Install Docker

Install NPM

http://lmgtfy.com/?q=install+npm+on+windows/linux/mac
Install Docker | Docker Documentation

Download | Node.js

Install Node / NPM

$ brew install node

Validate Node and NPM

$ node -v 
$ npm -v

[x] Install NPM

Create Neo4j Docker container

So we have docker installed, NPM will be used later but now I want to get Neo4j up and running.

Neo4j (by version) Docker Script

Create executable and file
Paste the following docker run command into an executable file like: neo4j-docker.sh

if [ ! -d "$HOME/neo4j/databases/$1/" ]; then
	echo "Making directory structure"
  mkdir -p $HOME/neo4j/databases/$1/{data,plugins,import,logs,conf}
	echo "Downloading APOC"
curl -o $HOME/neo4j/databases/$1/plugins/apoc-3.3.0.1-all.jar https://github.com/neo4j-contrib/neo4j-apoc-procedures/releases/download/3.3.0.1/apoc-3.3.0.1-all.jar
	echo "Docker Run"
fi

docker run \
--publish=7474:7474 \
--publish=7687:7687 \
--volume=$HOME/neo4j/databases/$1/data:/data \
--volume=$HOME/neo4j/databases/$1/plugins:/plugins \
--volume=$HOME/neo4j/databases/$1/import:/import \
--volume=$HOME/neo4j/databases/$1/logs:/logs \
--env=NEO4J_dbms_memory_pagecache_size=2G \
--env=NEO4J_dbms_memory_heap_maxSize=4096m \
--env=NEO4J_AUTH=none \
--env=NEO4J_dbms_security_procedures_unrestricted=apoc.\\\* \
--env=NEO4J_ACCEPT_LICENSE_AGREEMENT=yes \
--ulimit=nofile=40000:40000 \
neo4j:$1-enterprise

So what does it do?

If the version directory doesn’t exist:
- Create the version folder in $HOME/neo4j/databases/
- Create folders to be mapped to the local machine
- Download APOC into the plugins folder
Download and run a Docker container and start Neo4j
- Localhost ports 7474 & 7687 are mapped to your the container so you can get access to:
  - Browser/HTTP (7474)
    - go check it out: http://localhost:7474
  - BOLT (7687)

It’s best practise to use either env variables or map the config directory
Add the following line in and remove all —env lines.

--volume=$HOME/neo4j/databases/$1/import:/import \

Change executable permissions

$ chmod +x neo4j-docker.sh

So we can create different docker containers with different versions of Neo4j by running a command similar to ./neo4j-docker.sh

Start Neo4j Docker Container

$ ./neo4j-docker.sh 3.3.2

Head over to http://localhost:7474 and create some data:

CREATE (p:Person{ name: "Luke"}),
       (p2:Person {name:"Mel"}),
       (p)-[pris_m:IS_MARRIED_TO]->(p2)

Test our query we want to use later:

MATCH (p:Person{ name: "Luke"})-[pris_m:IS_MARRIED_TO]->(p2)
RETURN p.name, pris_m.since

[x] Create Neo4j Docker container

Install SAM Local (beta)

Install SAM with NPM

$ npm install -g aws-sam-local

Verify the installation worked:

$ sam --version

If you get a permission error when using NPM (such as EACCES: permission denied), please see the instructions on this page of the NPM documentation: https://docs.npmjs.com/getting-started/fixing-npm-permissions.

Upgrading via npm

To update SAM once installed via NPM:

$ npm update -g aws-sam-local

[x] Install SAM Local (beta)

Container Talk - Docker Network

In order to get our containers (SAM Local and Neo4j) speaking to each other, we need to create a network which they will share:

Create a network

docker network create {network-name}

Now let's connect the neo4j we started running before to the network.
We'll need it's container id or name:

Get the container ID or name

docker ps

Use the network you created and the neo4j container name

Connect the Neo4j container to the network

docker network connect {network-name} {neo4j-container-id/name}

Now lets find the IP address for the neo4j container, use that in your code.

Get Neo4j container IP

docker inspect {neo4j-container-id}

At the bottom of the page you’ll find within the Array of JSON objects, a “NetworkSettings” object, you’ll want NetworkSettings.Networks.{network-name}

[x] Docker Network to allow containers to talk to each other

Create template.yaml

What does a template.yaml file need?

[ ] Create template.yaml

Python

AWSTemplateFormatVersion : '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: My first serverless application.
Resources:
  GraphHandler:
    Type: AWS::Serverless::Function
    Properties:
      Handler: main.lambda_handler
      Runtime: python3.6
      CodeUri: .
      Timeout: 10

- [ ] Python

JavaScript

AWSTemplateFormatVersion : '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: My first serverless application.
Resources:
  GraphHandler:
    Type: AWS::Serverless::Function
    Properties:
      Handler: index.handler
      Runtime: nodejs6.10
      CodeUri: .
      Timeout: 10

- [ ] JavaScript

Java

AWSTemplateFormatVersion : '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Writing my first serverless application.
Resources:
  GraphHandler:
    Type: AWS::Serverless::Function
    Properties:
      Handler: com.nineteen.neo4j.GraphHandler
      CodeUri: ./target/lambda-1.0-jar-with-dependencies.jar
      Runtime: java8
      Timeout: 999

Note: you have to provide the path of the compiled jar.
- [ ] Java

dotNetCore

Unfortunately we’re waiting for the SAM team to support .NetCore, sorry Chris Skardon :sadparrot:

Create Test Handler

Python

Within main.py paste the following:

import null;

#TODO: Pad this out a bit
def lambda_handler(event, context):
    return ""

- [ ] Python

JavaScript

'use strict';

exports.handler = (event, context, callback) => {
    console.log(event);
}

Props to @AdamCowley’s and his mad javascript skills. I don’t like being tortured by languages.
- [ ] JavaScript

Java

package com.nineteen.neo4j.example;

import com.amazonaws.services.lambda.runtime.Context;

private Driver driver;

public class Hello {

    public String myHandler(String name, Context context) {

        System.out.println("log data to cloudwatch from stdout");
        System.err.println("log data tp cloudwatch from stderr");
        return String.format("Hello %s.", name);
    }
}

- [ ] Java

[ ] Create test handlers

Invoke driver

Java

package com.nineteen.neo4j.example;

import com.amazonaws.services.lambda.runtime.Context;

public class Hello {
	private Driver driver;
	public String myHandler(String name, Context context) {
		if (driver == null)
		{
			driver = GraphDatabase.driver("{URL_OF_NEO4J_CONTAINER}", AuthTokens.basic("neo4j", ""));
		}
    }
}

- [ ] Java

[ ] Invoke Driver

Talk about possible problems and how expensive it is to invoke drives?

Run read query

[ ] Run read query

Python

JavaScript

'use strict';



exports.handler = (event, context, callback) => {
    const neo4j = require('neo4j-driver').v1;
    const driver = new neo4j.driver('bolt://172.17.0.2:7687', neo4j.auth.basic('neo4j', 'neo') );

    const session = driver.session();

    session.run(`MATCH (p:Person{ id: {id} }),
                (p)-[prlive:LIVES_AT]->(a:Address),
                (p)-[pris_p:IS_PARTNER_OF]->(p2:Person),
                (p2)-[p2ris_i:IS_IDENTIFIED_BY]->(g:GovID)
        WHERE p.name= {p_name} AND prlive.since> {prlive_since} AND a.addressLine1 = {addressLine1} AND p2.name = {p2_name}
        RETURN p.name, a.postCode, p2.name as p2name, g.type`, params)
        .then(res => {
            console.log(res.records.length);

            return res.records.map(row => {
                return [ row.get('p.name'), row.get('a.postCode'), row.get('p2name'), row.get('g.type') ];
            });
        })
        .then(json => {
            session.close();

            return json;
        })
        .then(json => {
            console.log(json);

            const end = new Date().getTime();

            console.log('-- END',  end);
            console.log('-- END', end - start);

            callback(null, {
                statusCode: 200,
                headers: { "x-custom-header" : "my custom header value" },
                body: json
            });
        })
        .catch(e => {
            callback(e);
        })
        .then(() => {
            return driver.close();
        });

}

## Java

package com.nineteen.neo4j.example;

import com.amazonaws.services.lambda.runtime.Context;

public class Hello {
	private Driver driver;
	public String myHandler(String name, Context context) {
		if (driver == null)
		{
			driver = GraphDatabase.driver("{URL_OF_NEO4J_CONTAINER}", AuthTokens.basic("neo4j", ""));
		}

		try (Session session = driver.session(AccessMode.READ))
      {
			sr = session.run(query);
			List resultList = sr.list();
			//TODO: add return: return resultList;
        }
        catch (ClientException ce)
        {
            ce.printStackTrace();
        }

      //return String.format("Hello %s.", name);
    }
}

Link to driver pages and different queries and transactions

Why we won’t close off the connection in case our lambda is reused

[ ] Why we won’t close off the connection in case our lambda is reused

Test Lambda function

[ ] Test Lambda function

Didn’t even break a sweat. BIG SHAQ - MANS NOT HOT

Links