/ Neo4j

Testing your Amazon Serverless Application Model (SAM) locally with Neo4j, Lambda and Docker

Testing your Amazon Serverless Application Model (SAM) locally with Neo4j, Lambda and Docker

So this is what I want to achieve. You should be able to do the same by the end of the post:

  • [ ] Install Docker
  • [ ] Install NPM
  • [ ] Create Neo4j Docker container
  • [ ] Docker Network to allow containers to talk to each other
  • [ ] Install SAM Local (beta)
  • [ ] Create template.yaml
    • [ ] Python
    • [ ] JavaScript
    • [ ] Java
  • [ ] Create test handler
    • [ ] Invoke driver
      • [ ] Talk about possible problems and how expensive it is to invoke drives?
    • [ ] Run read query
      • [ ] Link to driver pages and different queries and transactions
      • [ ] Why we won’t close off the connection in case our lambda is reused
  • [ ] Test Lambda function

Install Docker

http://lmgtfy.com/?q=install+docker+on+windows/linux/mac

Install Docker | Docker Documentation

If you’re on a mac:

  1. cmd+space to bring up Spotlight Search, type terminal and press enter

  2. Install Homebrew

$ /usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
  1. Update Homebrew
$ brew update
  1. Install Docker via Homebrew
$ brew cask install docker
  1. cmd+space to bring up Spotlight Search, type docker and press enter
    • When it starts up, it’ll create everything in the following location
$ ls -l /usr/local/bin/docker*
lrwxr-xr-x  1 laexample  domain Users  67 Apr 12 14:14 /usr/local/bin/docker -> /Users/laexample/Library/Group Containers/group.com.docker/bin/docker
lrwxr-xr-x  1 laexample  domain Users  75 Apr 12 14:14 /usr/local/bin/docker-compose -> /Users/laexample/Library/Group Containers/group.com.docker/bin/docker-compose
lrwxr-xr-x  1 laexample  domain Users  90 Apr 12 14:14 /usr/local/bin/docker-credential-osxkeychain -> /Users/laexample/Library/Group Containers/group.com.docker/bin/docker-credential-osxkeychain
lrwxr-xr-x  1 laexample  domain Users  75 Apr 12 14:14 /usr/local/bin/docker-machine -> /Users/laexample/Library/Group Containers/group.com.docker/bin/docker-machine
  1. Validate version
$ docker version
  • [x] Install Docker

Install NPM

http://lmgtfy.com/?q=install+npm+on+windows/linux/mac
Install Docker | Docker Documentation

Download | Node.js

  1. Install Node / NPM
$ brew install node
  1. Validate Node and NPM
$ node -v 
$ npm -v
  • [x] Install NPM

Create Neo4j Docker container

So we have docker installed, NPM will be used later but now I want to get Neo4j up and running.

Neo4j (by version) Docker Script

  1. Create executable and file
    Paste the following docker run command into an executable file like: neo4j-docker.sh
if [ ! -d "$HOME/neo4j/databases/$1/" ]; then
	echo "Making directory structure"
  mkdir -p $HOME/neo4j/databases/$1/{data,plugins,import,logs,conf}
	echo "Downloading APOC"
curl -o $HOME/neo4j/databases/$1/plugins/apoc-3.3.0.1-all.jar https://github.com/neo4j-contrib/neo4j-apoc-procedures/releases/download/3.3.0.1/apoc-3.3.0.1-all.jar
	echo "Docker Run"
fi

docker run \
--publish=7474:7474 \
--publish=7687:7687 \
--volume=$HOME/neo4j/databases/$1/data:/data \
--volume=$HOME/neo4j/databases/$1/plugins:/plugins \
--volume=$HOME/neo4j/databases/$1/import:/import \
--volume=$HOME/neo4j/databases/$1/logs:/logs \
--env=NEO4J_dbms_memory_pagecache_size=2G \
--env=NEO4J_dbms_memory_heap_maxSize=4096m \
--env=NEO4J_AUTH=none \
--env=NEO4J_dbms_security_procedures_unrestricted=apoc.\\\* \
--env=NEO4J_ACCEPT_LICENSE_AGREEMENT=yes \
--ulimit=nofile=40000:40000 \
neo4j:$1-enterprise

So what does it do?

  • If the version directory doesn’t exist:
    • Create the version folder in $HOME/neo4j/databases/
    • Create folders to be mapped to the local machine
    • Download APOC into the plugins folder
  • Download and run a Docker container and start Neo4j
    • Localhost ports 7474 & 7687 are mapped to your the container so you can get access to:

It’s best practise to use either env variables or map the config directory
Add the following line in and remove all —env lines.

--volume=$HOME/neo4j/databases/$1/import:/import \
  1. Change executable permissions
$ chmod +x neo4j-docker.sh

So we can create different docker containers with different versions of Neo4j by running a command similar to ./neo4j-docker.sh

  1. Start Neo4j Docker Container
$ ./neo4j-docker.sh 3.3.2
  1. Head over to http://localhost:7474 and create some data:
CREATE (p:Person{ name: "Luke"}),
       (p2:Person {name:"Mel"}),
       (p)-[pris_m:IS_MARRIED_TO]->(p2)
  1. Test our query we want to use later:
MATCH (p:Person{ name: "Luke"})-[pris_m:IS_MARRIED_TO]->(p2)
RETURN p.name, pris_m.since
  • [x] Create Neo4j Docker container

Install SAM Local (beta)

  1. Install SAM with NPM
$ npm install -g aws-sam-local

Verify the installation worked:

$ sam --version

If you get a permission error when using NPM (such as EACCES: permission denied), please see the instructions on this page of the NPM documentation: https://docs.npmjs.com/getting-started/fixing-npm-permissions.

Upgrading via npm

To update SAM once installed via NPM:

$ npm update -g aws-sam-local
  • [x] Install SAM Local (beta)

Container Talk - Docker Network

In order to get our containers (SAM Local and Neo4j) speaking to each other, we need to create a network which they will share:

  1. Create a network
docker network create {network-name}

Now let's connect the neo4j we started running before to the network.
We'll need it's container id or name:

  1. Get the container ID or name
docker ps

Use the network you created and the neo4j container name

  1. Connect the Neo4j container to the network
docker network connect {network-name} {neo4j-container-id/name}

Now lets find the IP address for the neo4j container, use that in your code.

  1. Get Neo4j container IP
docker inspect {neo4j-container-id}

At the bottom of the page you’ll find within the Array of JSON objects, a “NetworkSettings” object, you’ll want NetworkSettings.Networks.{network-name}

  • [x] Docker Network to allow containers to talk to each other

Create template.yaml

What does a template.yaml file need?

  • [ ] Create template.yaml

Python

AWSTemplateFormatVersion : '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: My first serverless application.
Resources:
  GraphHandler:
    Type: AWS::Serverless::Function
    Properties:
      Handler: main.lambda_handler
      Runtime: python3.6
      CodeUri: .
      Timeout: 10

- [ ] Python

JavaScript

AWSTemplateFormatVersion : '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: My first serverless application.
Resources:
  GraphHandler:
    Type: AWS::Serverless::Function
    Properties:
      Handler: index.handler
      Runtime: nodejs6.10
      CodeUri: .
      Timeout: 10

- [ ] JavaScript

Java

AWSTemplateFormatVersion : '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Writing my first serverless application.
Resources:
  GraphHandler:
    Type: AWS::Serverless::Function
    Properties:
      Handler: com.nineteen.neo4j.GraphHandler
      CodeUri: ./target/lambda-1.0-jar-with-dependencies.jar
      Runtime: java8
      Timeout: 999

Note: you have to provide the path of the compiled jar.
- [ ] Java

dotNetCore

Unfortunately we’re waiting for the SAM team to support .NetCore, sorry Chris Skardon :sadparrot:

Create Test Handler

Python

Within main.py paste the following:

import null;

#TODO: Pad this out a bit
def lambda_handler(event, context):
    return ""

- [ ] Python

JavaScript

'use strict';

exports.handler = (event, context, callback) => {
    console.log(event);
}

Props to @AdamCowley’s and his mad javascript skills. I don’t like being tortured by languages.
- [ ] JavaScript

Java

package com.nineteen.neo4j.example;

import com.amazonaws.services.lambda.runtime.Context;

private Driver driver;

public class Hello {

    public String myHandler(String name, Context context) {

        System.out.println("log data to cloudwatch from stdout");
        System.err.println("log data tp cloudwatch from stderr");
        return String.format("Hello %s.", name);
    }
}  

- [ ] Java
  • [ ] Create test handlers

Invoke driver

Java

package com.nineteen.neo4j.example;

import com.amazonaws.services.lambda.runtime.Context;

public class Hello {
	private Driver driver;
	public String myHandler(String name, Context context) {
		if (driver == null)
		{
			driver = GraphDatabase.driver("{URL_OF_NEO4J_CONTAINER}", AuthTokens.basic("neo4j", ""));
		}
    }
}  

- [ ] Java
  • [ ] Invoke Driver

Talk about possible problems and how expensive it is to invoke drives?

Run read query

  • [ ] Run read query

Python

JavaScript

'use strict';



exports.handler = (event, context, callback) => {
    const neo4j = require('neo4j-driver').v1;
    const driver = new neo4j.driver('bolt://172.17.0.2:7687', neo4j.auth.basic('neo4j', 'neo') );

    const session = driver.session();

    session.run(`MATCH (p:Person{ id: {id} }),
                (p)-[prlive:LIVES_AT]->(a:Address),
                (p)-[pris_p:IS_PARTNER_OF]->(p2:Person),
                (p2)-[p2ris_i:IS_IDENTIFIED_BY]->(g:GovID)
        WHERE p.name= {p_name} AND prlive.since> {prlive_since} AND a.addressLine1 = {addressLine1} AND p2.name = {p2_name}
        RETURN p.name, a.postCode, p2.name as p2name, g.type`, params)
        .then(res => {
            console.log(res.records.length);

            return res.records.map(row => {
                return [ row.get('p.name'), row.get('a.postCode'), row.get('p2name'), row.get('g.type') ];
            });
        })
        .then(json => {
            session.close();

            return json;
        })
        .then(json => {
            console.log(json);

            const end = new Date().getTime();

            console.log('-- END',  end);
            console.log('-- END', end - start);

            callback(null, {
                statusCode: 200,
                headers: { "x-custom-header" : "my custom header value" },
                body: json
            });
        })
        .catch(e => {
            callback(e);
        })
        .then(() => {
            return driver.close();
        });

}

## Java

package com.nineteen.neo4j.example;

import com.amazonaws.services.lambda.runtime.Context;

public class Hello {
	private Driver driver;
	public String myHandler(String name, Context context) {
		if (driver == null)
		{
			driver = GraphDatabase.driver("{URL_OF_NEO4J_CONTAINER}", AuthTokens.basic("neo4j", ""));
		}

		try (Session session = driver.session(AccessMode.READ))
      {
			sr = session.run(query);
			List resultList = sr.list();
			//TODO: add return: return resultList;
        }
        catch (ClientException ce)
        {
            ce.printStackTrace();
        }

      //return String.format("Hello %s.", name);
    }
}

Link to driver pages and different queries and transactions

Why we won’t close off the connection in case our lambda is reused

  • [ ] Why we won’t close off the connection in case our lambda is reused

Test Lambda function

  • [ ] Test Lambda function

Didn’t even break a sweat. BIG SHAQ - MANS NOT HOT

Links