mkdocs include

markdown-include is a pip package that can be use with mkdocs.

Install the python package markdown-include per https://github.com/cmacmackin/markdown-include
Add the following lines in your mkdocs. Set the base_path to your choice.

markdown_extensions:
  - markdown_include.include:
      base_path: .

And finally add the contents of one file into another by just doing

{!yourfile!}

For real example, check out the this writeup in github

mkdocs themes

Find your favorite theme https://github.com/mkdocs/mkdocs/wiki/MkDocs-Themes
pip install that theme
add in mkdocs as follows

theme: material

Local Dynamodb (docker)

#  Setting up dynamodb locally (docker)

References

docker: https://hub.docker.com/r/amazon/dynamodb-local/
aws cli: https://docs.aws.amazon.com/cli/latest/reference/dynamodb/index.html
https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html

Pull and run docker on port 7000

docker pull amazon/dynamodb-local docker run -p 7000:8000 amazon/dynamodb-local

$ cat ~/.aws/credentials

[default]
aws_access_key_id = fakeid
aws_secret_access_key = fakekey

$ cat ~/.aws/config

[default]
region = x

[default] aws_access_key_id = fakeMyKeyId aws_secret_access_key = fakeSecret

AWS CLI Commands

List Tables aws dynamodb list-tables --endpoint-url http://localhost:7000

Create a table

KeyType - The role of the attribute: HASH - partition key RANGE - sort key

1- Creating a table MusicCollection with partition field 'Artist' and sort field 'SongTitle'

aws dynamodb create-table --table-name MusicCollection --attribute-definitions AttributeName=Artist,AttributeType=S AttributeName=SongTitle,AttributeType=S --key-schema AttributeName=Artist,KeyType=HASH AttributeName=SongTitle,KeyType=RANGE --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5 --endpoint-url http://localhost:7000

2- Creating a table Computers with partition field 'Vendor' only

aws dynamodb create-table --table-name Computers --attribute-definitions AttributeName=Vendor,AttributeType=S --key-schema AttributeName=Vendor,KeyType=HASH  --provisioned-throughput ReadCapacityUnits=5,WriteCapacityUnits=5 --endpoint-url http://localhost:7000

Table details

aws dynamodb describe-table --table-name Computers --endpoint-url http://localhost:7000

Dynamodb Go SDK

$ cat main.go $ go run main.go


package main

import (
  "fmt"

  "github.com/aws/aws-sdk-go/aws"
  "github.com/aws/aws-sdk-go/aws/session"
  "github.com/aws/aws-sdk-go/service/dynamodb"
)

const (
  tablename = "Blogs"
)

func main() {

  ////////////////////////// create table ////////////////////////////////////
  ////////////////////////////////////////////////////////////////////////////

  // creating a session with Dynamodb
  awsConfig := aws.Config{
    Endpoint: aws.String("http://localhost:7000"),
  }

  sess := session.Must(session.NewSessionWithOptions(session.Options{
    SharedConfigState: session.SharedConfigEnable,
    Config:            awsConfig,
  }))

  // Create dynamodb service using the above created session
  svc := dynamodb.New(sess, aws.NewConfig())

  // Create table schema
  input := &dynamodb.CreateTableInput{
    AttributeDefinitions: []*dynamodb.AttributeDefinition{
      {
        AttributeName: aws.String("Title"),
        AttributeType: aws.String("S"),
      },
    },
    KeySchema: []*dynamodb.KeySchemaElement{
      {
        AttributeName: aws.String("Title"),
        KeyType:       aws.String("HASH"),
      },
    },
    ProvisionedThroughput: &dynamodb.ProvisionedThroughput{
      ReadCapacityUnits:  aws.Int64(5),
      WriteCapacityUnits: aws.Int64(5),
    },
    TableName: aws.String(tablename),
  }

  result, err := svc.CreateTable(input)
  if err != nil {
    fmt.Println("-------- -----------")
    fmt.Println(err)
  }

  fmt.Println("+++++++++ 1 ++++++++")
  fmt.Println(sess)
  fmt.Println(result)

  ////////////////////////// list tables ////////////////////////////////////
  ////////////////////////////////////////////////////////////////////////////

  listInput := &dynamodb.ListTablesInput{}
  tables, err := svc.ListTables(listInput)
  fmt.Println("+++++++++ 2 ++++++++")
  fmt.Println(tables)

  ////////////////////////// insert item ////////////////////////////////////
  ////////////////////////////////////////////////////////////////////////////

  putInput := &dynamodb.PutItemInput{
    Item: map[string]*dynamodb.AttributeValue{
      "Title": {
        S: aws.String("No One You Know22"),
      },
    },
    ReturnConsumedCapacity: aws.String("TOTAL"),
    TableName:              aws.String(tablename),
  }

  item, err := svc.PutItem(putInput)
  fmt.Println("+++++++++ 3 ++++++++")
  fmt.Println(item, err)
  ////////////////////////// read all items  ////////////////////////////////////
  ////////////////////////////////////////////////////////////////////////////

  params := &dynamodb.ScanInput{
    TableName: aws.String(tablename),
  }
  scanResult, err := svc.Scan(params)
  if err != nil {
    fmt.Errorf("failed to make Query API call, %v", err)
  }

  fmt.Println("+++++++++ 4 ++++++++")
  fmt.Println(scanResult)

  ////////////////////////// read an item  ////////////////////////////////////
  ////////////////////////////////////////////////////////////////////////////

  getInput := &dynamodb.GetItemInput{
    Key: map[string]*dynamodb.AttributeValue{
      "Title": {
        S: aws.String("No One You Know"),
      },
    },
    TableName: aws.String(tablename),
  }

  itemQ, err := svc.GetItem(getInput)
  fmt.Println("+++++++++ 5 ++++++++")
  fmt.Println(itemQ)
  ////////////////////////// delete item  ////////////////////////////////////
  ////////////////////////////////////////////////////////////////////////////

  delItemInput := &dynamodb.DeleteItemInput{
    Key: map[string]*dynamodb.AttributeValue{
      "Title": {
        S: aws.String("No One You Know"),
      },
    },
    TableName: aws.String(tablename),
  }
  delItem, err := svc.DeleteItem(delItemInput)
  fmt.Println("+++++++++ 6 ++++++++")
  fmt.Println(delItem, err)

  ////////////////////////// update item  ////////////////////////////////////
  ////////////////////////////////////////////////////////////////////////////



var params = {
    TableName:tablename,
    Key:{
        "Title": "No One You Know"
    },
    UpdateExpression: "set info.rating = :r, info.plot=:p, info.actors=:a",
    ExpressionAttributeValues:{
        ":r":5.5,
        ":p":"Everything happens all at once.",
        ":a":["Larry", "Moe", "Curly"]
    },
    ReturnValues:"UPDATED_NEW"
};
  ////////////////////////// delete table  ////////////////////////////////////
  ////////////////////////////////////////////////////////////////////////////
  delTabInput := &dynamodb.DeleteTableInput{
    TableName: aws.String(tablename),
  }
  delTable, err := svc.DeleteTable(delTabInput)

  fmt.Println("+++++++++ 7 ++++++++")
  fmt.Println(delTable, err)
}

Bug Fix

Note, if you find error like "table x does not exist", then run dynamodb with sharedb turned on.

docker run -p 8000:8000 -v $(pwd)/local/dynamodb:/data/ amazon/dynamodb-local -jar DynamoDBLocal.jar -sharedDb -dbPath /data

https://forums.aws.amazon.com/message.jspa?messageID=869395

go json advanced

## `UseCase`:

Have a known given set of json keys say wantKeys. wantKeys := []string{"name", "age"} . Task is to check keys in a target object against "wantKeys". If they are same, it means received data is per our expectation, otherwise we got unexpected data.

givenKeys := []string{"name", "age"}
    var receivedKeys []string

    // Assume the data we are getting is in []byte format. Create a test data. 

    testDataByte := []byte(`{"name": "john", "age":55}`)


    // Note: We expect our data may be different, so we can't define struct with fieldnames and unmarshal. 
    // Let's use empty interface to deal our arbitrary data.

    var data interface{}

    json.Unmarshal(testDataByte, &data)

    // Actual fun starts! 
    // How do I get all the keys from this data to compare with given keys?

    // Use type assertion to access data's map[string]interface{}

    m := data.(map[string]interface{}) 

    for k, _ := range m {
      receivedKeys = append(receivedKeys, k)
    }

    if len(givenKeys) != len(receivedKeys) {
      fmt.Printf("got number of keys in response %d, but want %d", len(receivedKeys), len(givenKeys))
    }

    sort.Strings(givenKeys)
    sort.Strings(receivedKeys)

    if sameKeys := reflect.DeepEqual(givenKeys, receivedKeys); !sameKeys {
          fmt.Printf("got keys in response %s, but want %s", receivedKeys, givenKeys)
    }

    fmt.Println("Yes receivedKeys and givenKeys are exactly same!")

Go Playground link https://play.golang.org/p/uByO3tR1qs9

Reference - https://blog.golang.org/json-and-go

go marshal & unmarshal

## `JSON Marshalling and Unmarshalling`

In this article we'll see

how to convert string data into bytes (marshal)
how to convert bytes into Go struct (unmarshal)
how to convert bytes into Go interface (unmarshal)

data into bytes

We coverted data into bytes in two ways:

One, using byte() function - read function convertToBytes
Other, using json.Marshal() - read function convertToBytesUsingJsonMarshal

bytes to struct

first defined a empty struct to hold values jb := Response{}
then use json.Unmarshal to deserliaze into this struct json.Unmarshal(<raw1 or raw2>,&jb)

bytes to interface

first defined a empty interface to hold values var jbi interface{}
then use json.Unmarshal to deserliaze into this interface json.Unmarshal(<raw1 or raw2>,&jbi)

Q: When to unmarshal in interface instead of struct

A: There could be many use cases. But one we found is field extraction. For example, if you are given a json with certain fields and you want to validate this json, meaning you want to make sure if it has all the required fields. SO first you need to extract all the fields right? And then match against a required fields set. Unlike Javascript we don't have the luxury to get the keys of an object by doing Object.keys() to get all keys :( So what we do? This is covered in go-json-advanced writeup.

Go Playground link https://play.golang.org/p/Fia-l18hWvi

Keys	Action
`?`	Open this help
`n`	Next page
`p`	Previous page
`s`	Search