Recently I have been picking up Terraform on the job. I love the idea of Infrastructure as Code where all the resources are described in files that are versioned like regular code.
In this post, I am attempting to do a brain dump of what I have learned so far.
In short, I would describe Terraform as an IAC tool that lets you author files (.tf extension) that describes your infrastructure and helps you to manage (apply, destroy) the infrastructure state.
Read more...A few weeks ago, I started tinkering with our government’s HDB resale prices API . I had an idea to use recent transaction data to gauge if a listing on a property portal (e.g., property guru) is overpriced or fairly priced. Paralyzed by the sheer number of variables, I soon lost steam.
Resuming after a two week hiatus, I scoped the coding down to just fetching recent transactions which matched a listing’s address (specific block and street), flat type (4 room or 5 room) and floor level (low, mid and high).
Read more...I extracted the JSON utility that I wrote previously into a standalone repository. Cleaned up the code and tests!
Check it out here!
I set myself a challenge to implement an API endpoint in Common Lisp, in hope of putting to use a few things that I have learnt recently,
cl-project:make-projecthunchentoot web server libraryrove testing framework to test drive implementationAnd so, I settled on coding up a mortgage calculator which takes in principal (loan amount), tenure (in years) and annual interest rate, and returns a list of monthly payments, each comprising of interest, principal repayment, and outstanding principal so far.
Read more...I extracted all the boilerplate to perform a REST get operation into rest-helpers:
(defun get-json (url)
(let ((json))
(ignore-errors
(multiple-value-bind
(data status)
(drakma:http-request url)
(if (= 200 status)
(let* ((str (flexi-streams:octets-to-string data)))
(setq json (json:decode-json-from-string str))))))
(unless json (format t "~&Unable to fetch from ~A~%" url))
json))
Using both rest-helpers and json-helpers (from previous post), to get a list of astronauts in space requires much less code:
I wrote a helper get-json-value to extract values from json output produced by cl-json.
(defun property-p (property)
(funcall #'consp property))
(defun get-property-value (property)
(funcall #'cdr property))
(defun get-property-key (property)
(funcall #'car property))
(defun json-object-p (json)
(and (listp json) (every #'consp json)))
(defun json-array-p (json)
(and (listp json) (every json-object-p json)))
(defun json-p (json)
(or (json-object-p json) (json-array-p json)))
(defun get-json-value (keys json)
(princ (list keys json))
(terpri)
(cond
((null keys) json)
((json-p json)
(let ((key (car keys)))
(if (numberp key)
(get-json-value (cdr keys) (nth key json))
(let ((property
(find-if
#'(lambda (property)
(and (property-p property)
(eq (get-property-key property) (car keys)))) json)))
(if property
(get-json-value (cdr keys) (get-property-value property)))))))
(t nil)))
For example, a json object produced by cl-json looks something like this:
Read more...This weekend, I learnt how to fetch and parse JSON using the drakma HTTP client and cl-json libraries. And some simple error handling as well.
Tests:
(ql:quickload :testlib)
(use-package :testlib)
(run (context "who is in space?"
(context "people-p"
(context "when key is people"
(test "it should return true" (people-p (cons :people '()))))
(context "when key is not people"
(test "it should return false" (not (people-p (cons :foo '()))))))
(context "get-name"
(context "when name is present"
(test "it should return name" (string= (get-name `(,(cons :name "foo"))) "foo")))
(context "when name is not present"
(test "it should return nil" (eq nil (get-name `(,(cons :craft "bar")))))))
(context "get-craft"
(context "when craft is present"
(test "it should return craft" (string= (get-craft `(,(cons :craft "foo"))) "foo")))
(context "when craft is not present"
(test "it should return nil" (eq nil (get-craft `(,(cons :name "bar")))))))
(context "format-astronaut"
(test "it should format correctly"
(string= (format-astronaut `(,(cons :name "foo") ,(cons :craft "iss")) nil) (format nil "~&foo (iss)~%"))))
(context "get-people"
(test "it should return people list from property pair"
(equal (get-people (cons :people '(1 2 3))) '(1 2 3))))))
Implementation:
Read more...Writing a CSV parser is a good way to learn about streams and string formatting in Common Lisp. The goal is to take any CSV file and pretty print the contents. This is done by setting the width of each column to the widest value in that column + 1.
Code for the CSV parser:
(defun get-lines (input-stream)
(let ((lines '()))
(do ((line (read-line input-stream nil nil)
(read-line input-stream nil nil)))
((null line) (nreverse lines))
(push line lines))))
(defun get-columns (line)
(let ((columns '()))
(do ((i 0 (+ j 1))
(j (position #\, line :start 0) (position #\, line :start (+ j 1))))
((null j) (progn (push (subseq line i) columns) (nreverse columns)))
(push (subseq line i j) columns))))
(defun get-rows (lines)
(map 'list #'get-columns lines))
(defun get-fitting-column-width (rows n)
(let ((get-nth-column-width #'(lambda (row) (length (nth n row)))))
(+ (apply #'max (map 'list get-nth-column-width rows)) 1)))
(defun get-fitting-column-widths (rows)
(let ((cols (apply #'max (map 'list #'length rows))))
(if (zerop cols)
'()
(loop as n from 0 to (- cols 1) collect (get-fitting-column-width rows n)))))
(defun format-rows (rows &optional (output-stream t))
(let* ((widths (get-fitting-column-widths rows))
(format-row
(lambda (row)
(let ((n 0))
(map 'list
#'(lambda (col)
(let ((w (nth n widths)))
(incf n)
(format nil (format nil "~~~AA" w) col))) row)))))
(format output-stream "~{~&~{~A~}~%~}" (map 'list format-row rows))))
(defun format-csv (input-stream &optional (output-stream t))
(format-rows (get-rows (get-lines input-stream)) output-stream))
And the test output of tests written using the little test library I have been refining:
Read more...The previous version of test library had a number of problems, most importantly it relies heavily on the lexical variables indent and outcomes shared across multiple test runs. This meant that if test execution was terminated halfway and restarted for a set of tests, the indent and outcomes might not have been correctly reset, hence resulting wrong test output formatting.
The fixed version:
(defvar *test-output-stream* t)
(defun make-spaces (n)
(make-string n :initial-element #\space))
(defun format-outcome (depth name outcome &optional (output-stream *test-output-stream*))
(format output-stream "~&~ATest `~A` ~:[failed~;passed~].~%" (make-spaces depth) name outcome))
(defun format-context (depth name &optional (output-stream *test-output-stream*))
(format output-stream "~&~A~A~%" (make-spaces depth) name))
(defun format-outcomes (outcomes &optional (output-stream *test-output-stream*))
(format output-stream "~&~A out of ~A tests passed~%" (count t outcomes) (length outcomes)))
(defmacro test (name predicate)
(let ((outcome (gensym)))
`(lambda (depth)
(let ((,outcome ,predicate))
(format-outcome depth ,name ,outcome) (list ,outcome)))))
(defmacro context (name &rest body)
`(lambda (depth)
(format-context depth ,name)
(let ((depth (+ depth 1)))
(reduce #'append (map 'list #'(lambda (f) (apply f `(,depth))) (list ,@body))))))
(defun run (root-context)
(format-outcomes (apply root-context `(,0))))
The newly fixed version removes dependency on shared indent and outcomes lexical variables, instead explicitly redefining a depth lexical variable and passing in depth to each test and context, hence leveraging the exit of corresponding lexical scopes to restore the correct depth to use for test output formatting. An outcome is returned from each test and context, resulting in an accumulated list of outcomes at the root level. A run function is also introduced to kickstart a root context, henceforth removing the need for checking depth to determine if outcomes should be printed.
Putting my test library to use, I tried to implement a simple password strength checker. I immediately realised that I needed a better way to organize my tests and test output. The current test output looks like this:
Test `it should return strong if it contains at least 1 lowercase alphabet, at least 1 uppercase alphabet, 1 special character, 1 digit and its length is at least 8` passed.
Test `it should return weak if it contains no lowercase alphabet even though it has at least 1 uppercase alphabet, 1 special character, 1 digit and its length is at least 8` passed.
Test `it should return weak if it contains no uppcase alphabet even though it has at least 1 lowercase alphabet, 1 special character, 1 digit and its length is at least 8` passed.
Test `it should return weak if it contains no special character even though it has at least 1 lowercase alphabet, 1 uppercase alphabet, 1 digit and its length is at least 8` passed.
Test `it should return weak if it contains no digit even though it has at least 1 lowercase alphabet, 1 uppercase alphabet, 1 special character and its length is at least 8` passed.
Test `it should return moderate if its length is less than 8 but at least 6 even though it has at least 1 lowercase alphabet, 1 uppercase alphabet, 1 special character, 1 digit` passed.
Test `it should return moderate if it contains at least 1 lowercase alphabet, 1 uppercase alphabet, 1 special character and its length is at least 6` passed.
Test `it should return weak if it contains no lowercase alphabet, even though it contains at least 1 uppercase alphabet, 1 special character and its length is at least 6` passed.
Test `it should return weak if it contains no uppcase alphabet even though it has at least 1 lowercase alphabet, 1 special character and its length is at least 6` passed.
Test `it should return weak if it contains no special character even though it has at least 1 lowercase alphabet, 1 uppercase alphabet and its length is at least 6` passed.
Test `it should return weak if its length is less than 6 even though it has at least 1 lowercase alphabet, 1 uppercase alphabet and 1 special character` passed.
11 out of 11 tests passed
Ideally, I want to be able to write tests with context and test, not unlike how other languages organize tests: