Serverless Architecture at the Small Scale

On Saturday, 16th September 2017, DDD Perth was held at the Perth Convention and Exhibition Centre.

DDD is one of the few regular software developer conventions in Perth, so it was very well attended with tickets selling out well before the day of the event. The concept is that it is organised by developers, for developers to attend and talk, and the subjects are selected democratically by the developer community, hence the name “DDD” — “Developer Developer Developer”.  There were six of us from Gaia Resources in attendance, including myself as an elected presenter, and there were some common threads to our responses to the day.

DJ0x5SwUMAEpumw

Ben in full flight at DDD 2017

Highlights

Unsurprisingly, the keynote and locknote speakers were well-received.

Gojko Adzic discussed in his keynote the challenges of developing and specifically controlling quality of cloud service based applications.  In particular, reliance on external services gives us some unique challenges compared to previous generations of software that were a lot more self-contained.  This leads to ideas such as testing in production, something that was previously frowned upon, and challenging our assumptions about data accuracy, especially as a result of automation gone wrong.

Kris Howard‘s locknote was about the social, rather than the technical, aspects of development, a trend that was present throughout the day with many non-technical topics being well-received.  Her concept of “leaving the tech industry better than we found it” aka “the campsite rule” highlighted the advantages of being a mentee or mentoring as a way of “giving back” and improving the tech industry, included useful suggestions on how to find and ask someone if they’d be willing to be your mentor.

Overall the day was well organised, well catered, and well run.  The talks were fairly diverse and informative, the staff and attendees were really friendly and inclusive, and you could tell that a lot of personal effort was put into making it a successful community event.  Of course, meeting people in the industry and catching up with other developers was a highlight. There was also a lot of genuine excitement about DDD itself and its growth and ongoing success.

Some other highlights for the team included Ian Hughes’ talk, “what a real sprint looks like”, Patima Tantiprasut’s talk about “Web, Wellness and Getting Sh*t Done”, Rami Ruhayel’s gentle introduction to neural networks, and Rob Crowley’s introduction to GraphQL.

IMG_20170916_085120

 Tony, Kehan and Shay captured “in the wild” at DDD

Serverless Architecture at the Small Scale

My talk was about “Serverless Architecture at the Small Scale”, and it turned out to be a popular topic, with the room overflowing with audience members – standing room only!  This was my first time talking at a conference, and it was quite an experience.  There is a lot more preparation work that goes into a presentation than you might expect; all up I spent about a week preparing, practising and polishing my talk!

Serverless is a somewhat misleading term; definition by absence is a difficult starting point.  I prefer the more natural “cloud-native”, which describes what these applications are rather than what they are not — one that extensively or exclusively uses cloud services, for data storage, data retrieval, and for business logic.  The last of these is possible due to “Function as a Service” (FaaS) such as AWS Lambda, Microsoft Azure Functions, and Google Cloud Functions.  This is a relatively new technology, available since only 2015, which allows you to run your own code without provisioning or maintaining servers.  The code is run inside a standardised container — hence the “serverless” moniker — and uses additional services to provide entry points.  This model fits particularly well with a very high proportion of the use cases that we encounter.

The “small scale” aspect is also very relevant to us at Gaia Resources.  Our work with citizen science groups, small collections, and researchers, is very much at the small scale, especially when compared to “large scale” users like Netflix, Target, Amazon, etc.  In fact a lot of the projects we deliver fit comfortably within the “perpetual free tier” offered by many of the cloud services.

We have used serverless technology on quite a few projects so far, including multiple survey applications (of the question-and-answer kind, as opposed to the biological survey kind), with a serverless API behind single-page application (SPA) frontends.  These projects have been delivered for clients such as the UWA Centre for the Built Environment and Health, BirdLife Australia, and the Clean Air Urban Landscapes hub.  Several other projects utilise serverless technology, including the Tasmanian Collection and the WA New Music Archive.

Demonstration

Photograph of the audience, processed through the demonstration application

The main part of my DDD talk consisted of a live demonstration, which despite some technical issues, was successful in the end.  In fact I think the audience appreciated my perseverance and eventually making it work, and I received a lot of positive comments afterwards, both in person and through the conference’s feedback system.  The demonstration used some very simple code and the Serverless framework to receive an image from an S3 bucket, normalise it and add a watermark, then save the image in a different S3 bucket.  An email was then generated, and any images were accessible via an API created using API Gateway.  There is a GitHub repository containing the code that I used, if you want to check it out.

The concepts were demonstrated using a photographic image, because it was visual and therefore interesting for the audience.  However this is a technique that we have used successfully with uploaded CSV data files, processing the data and storing it in a database or sending it to another API.  The code required is very minimal, delivering a great effort-to-reward ratio.  I hope to do more such presentations in the future.

Outcomes

The various talks presented during the day, and our own discussions with each other and developers outside of Gaia, have given us a lot of takeaway messages to consider.  Having the opportunity to hear some of Perth’s best developers talk about their experiences and recommendations is an excellent opportunity to get away from our day-to-day work and find some fresh perspective.

DDD Perth was a great experience and everyone from Gaia who participated, as well as many others that I spoke to on the day, are keen to participate in next year’s conference.  If you’d like to know more about the talk or what we do at Gaia, feel free to email me.

Ben

Leave a Reply