Not about school

thinking emoji

Consider this a parable of sorts for those building products. I’ll leave it up to the reader to determine its meaning.

Carl is a senior in high school and has a D in Algebra II heading into the last semester. He wants to bring his grade up to a B so that he can qualify for college scholarships. Quizzes and tests make up 80% of a student’s grade. The other 20% consists of completing homework (15%) and classroom work (5%). Carl’s Algebra II teacher, Mr. Smith, insists that Carl catch up on all his missing classroom work first, even though most of that work will not help on upcoming quizzes and tests. Mr. Smith works with his student to create a rather large yet well-defined backlog of all the classroom work Carl needs to catch up on so they can track his progress together. Carl diligently works on burning down that backlog. He finally catches up on the classroom work near the end of the school year. Mr. Smith pats him on the back. Carl’s final Algebra II grade is a D+.

A remote retrospective

It’s been a while since I last facilitated an all remote retrospective. Below is an email I sent out to the team I’m currently working with to help us prepare for our first “retro” together. We’ve since held the retro and it went well overall, so I thought this prep and guidance might be useful for others to iterate on.

—–

Hi All,

On Tuesday we’re going to have a Project Retrospective (aka “retro”). In my experience, retros are hugely beneficial opportunities for teams to learn and grow.

Please take the time to read the rest of this email. It outlines how the retro will run. In order to make the most of our time, it’s best to come prepared. :-)

Retro Goals

  • Learning jointly – From each other’s different perspectives, feelings, and current thoughts about where the team is at since launching the project.
  • Taking action Based on learning together, we identify where we can most benefit from improvement and take action.
  • Strengthening the team – We’re in this together. By listening, learning, and taking action on what we’ve learned, we develop a stronger bond that is bigger than “just the work”.

Ground Rules

Regardless of what we discover, we understand and truly believe that everyone did the best job he or she could, given what was known at the time, his or her skills and abilities, the resources available, and the situation at hand.

  1. Be respectful
  2. Be present (no phones, no “side chats/conversations”, no browsing, etc.)
  3. Everyone gets a turn to speak
  4. No interrupting
  5. No judgement on feedback (use “I statements“)

Before The Retro (between now and when we meet)

Please take time between now and when we meet to think about the following in relation to your experience and/or what you observed during your time with the TGE project:

  • What you want us to continue doing
  • What you’d like us to start doing
  • What you’d like us to stop doing

You can add your items to this doc, which we’ll use as part of the retro: Google Doc

The Start

Check-in: We’ll go around the call and have everyone provide one word for how they’re feeling in the moment. Have trouble coming up with a word for how you’re feeling? I know I do sometimes! Try this to help identify a word: https://verbaliststravel.files.wordpress.com/2016/01/the-language-and-vocabulary-wheel-for-feelings-verbalists.jpg

The Middle

Part 1) We’ll use the “Before the Retro” section above to guide the initial discussion. What we want to: continue, start, and stop doing. If you haven’t added your items to each area, we’ll have a short time for everyone to add to each list.

Part 2) We’ll go through each item and open it up for questions/clarification. I will do my best to encourage open, focused discussion that honors our time.

Part 3) Everyone gets 3 votes per area to put against the items they think we should take action on now, not later. You can apply more than 1 vote to an item if you feel that strongly about it.

Part 4) Identify the top item in each area and determine next steps (which can be as simple as identifying an owner to drive the action post-retro). …But what about the other items – aren’t they important too? Yes! But focus is key. Just because an item isn’t made “top priority” doesn’t mean it won’t see progress in the future.

I’m accountable for ensuring the 3 items we identified to take action on make progress. Don’t expect miracles. Some things require a a fair amount of determination over a period of time to show results. My commitment is to continue to provide visibility to the items and help move them forward. Expect this to be part of our weekly meetings, even if it’s just a quick update.

The End

Check-out: Everyone on the call gets a brief (30 seconds or less) opportunity to express closing thoughts/feelings now that the retro is at a close.

Looking forward to learning and improving with you all!

Josh

Deploying a Python Flask web app on AWS Lambda

The best server is no server?

OK, servers are still involved with “serverless computing“, but not ones that you and I need to worry about maintaining and scaling. While serverless platforms like Amazon’s AWS Lambda, Google’s Cloud Functions, IBM’s OpenWhisk, Microsoft’s Azure Functions, and others aren’t the right fit for every need, I’m beginning to think they’re applicable to more apps and services than not.

I’ve been playing around with AWS Lambda for a little while now. After getting a number of functions up and running, I started to think about how this platform might be applicable for web apps. While I’ve built a Slack app using Node.js, I have to admit I missed the elegance of Python.

I settled on getting a little web app based on a Python web framework up and running on Lambda. I wanted the deployment to be as painless as possible (i.e. easily repeatable once configured). For this exercise I didn’t care about integrating with outside services. I was fine with running against a local SQLite database, knowing I could switch over to RDS or another database at another time. Here is what I ended up selecting for this task:

I chose Flask because it’s well regarded and simple to get running. I chose the Serverless framework (versus something like Zappa) because Serverless provides the ability to deploy Python and non-Python based projects quickly while also allowing you to deploy to other cloud providers like Google, Microsoft, etc. I chose the Flaskr app because it’s simple but not too simple.

Before we get started

There’s quite a bit that needs to be setup and running before my howto below will work for you. Here is what I expect to be running:

Also, my instructions assume you’re on Linux or Mac. If you’re on Windows, you’ll need to adjust the commands for your environment.

How to get it all running

  1. Setup Serverless and the Flaskr app locally
  2. Modify SQLite code to run in Lambda (and locally)
  3. Configure the Serverless deployment
  4. Deploy to AWS
  5. Remove from AWS
1. Setup Serverless and the Flaskr app locally

In a terminal session run the following:

These commands install the Serverless framework, downloads the Flaskr app from Github, sets up the app to run with Flask, and initializes a local SQLite database with the necessary table. If all goes well, you should be able to start up Flaskr locally:

In this case I can access http://127.0.0.1:5000/ in my browser and see something like this:

2. Modify SQLite code to run in Lambda (and locally)

We need to modify the flaskr/flaskr.py file in order to get SQLite to work in Lambda. In case it wasn’t already obvious, this setup is not meant for production. With that out of the way…

Modify the replace the app.config.update section with the following to configure the SQLite database file to be created in /tmp:

flaskr/flaskr.py

Then look further down in the file and modify the @app.route(‘/’) section with the following:

The above change tries to query the database and if there’s an exception, will attempt to call our init_db() function to create the SQLite database and set an empty entries variable.

3. Configure the Serverless deployment

Now we need to setup our Serverless deployment by setting the serverless.yml file to:

serverless.yml

These settings tell AWS we have a Python based function to run, we’ll use the serverless-wsgi plugin, all http requests are to be handled by the WSGI handler to run our flaskr.app. Note we need to explicitly set events to respond for “/” even though we have the catch all entry of {proxy+} set. If we don’t, AWS API Gateway will return an error of Missing Authentication Token.

Let’s test this locally one last time before we deploy to AWS:

You should get a message telling you where the Flask app is running (e.g. http://localhost:5000) to the one above, complete with a locally running instance of the flaskr app.

4. Deploy to AWS

Now we’re ready to deploy to AWS:

This can take a little bit to run, as this single command takes care of all the AWS setup for us to access our flaskr app via Lambda. When successful, you should see output towards the bottom similar to the following:

Copy the URL from the ServiceEndpoint and access it in your web browser. You should see something familiar:

If you don’t get a response or see an error of some sort, it’s time to look at the logs on AWS:

If you want to make changes to the app (e.g. change templates, modify routes in flaskr/flaskr.py, etc.) and don’t need to make changes to your serverless.yml file, you can do a much quicker re-deploy of the code (i.e. no AWS configuration):

5. Remove from AWS

Once you’re done with these steps, it’s a good idea to remove your function so that you don’t get unexpectedly charged for anything related to this deployment. Don’t fret, it’s only one line to get it all running again. :-)

Where to from here?

I may get this example working with a proper database. If I do, I’ll be sure to add a new post explaining how I did that.

Silver bullets

silverbullet

In folklore, a bullet cast from silver is often the only weapon that is effective against a werewolf, witch, or other monsters.

https://en.wikipedia.org/wiki/Silver_bullet

I’m not sure how many of us in tech are hunting werewolves, witches, or monsters, but there are A LOT of us looking for silver bullets. Take one look around at the world of software development and operations today and it’s baffling. A perfect example of what I’m talking about can be found in this CircleCI blog post, which sounds like nonsense but actually sums up the current state of software development and operations quite nicely. Maybe we’re not looking for silver bullets. Maybe we’re looking for the gold bullet, then the platinum bullet, then we move onto missiles, bombs, etc. It never ends.

I’m not talking about the evolution of technology as a whole. We have a desire to always learn and progress. What we have in software development and operations today is a hyper spin cycle of the next shiny object. Very little of it gets mature enough before it’s thrown out in favor of the new-new silver bullet. Unlike the last silver bullet, this one will kill werewolves AND witches. If you’re a software engineer caught up in this never ending hype you can find yourself using all the latest and hottest tech with so much complexity it will melt the minds of mere mortals. This happens rather easily because the hottest tech often comes from companies like Google, Facebook, Netflix, etc. who are solving extremely large and complex problems. They are generously (and often strategically) open sourcing a lot of that work, which turns into the hottest tech of the month club. But most of us are not solving problems at that scale. When we take their tech and apply it to our problem we can easily find ourselves with a lot of moving parts when only a few would likely suffice for our problems. Multiply this problem by the output of the firehose of “innovation” happening. I put innovation in quotes because what comes to us in the guise of innovation is often a company’s hope to ever so slightly improve upon a good enough solution and take away market share in the process. Even when there is legit innovation, the benefits of adopting a brand new stack or even layer in the stack are questionable when weighed against building upon a solid foundation.

Again, to get the best grasp of what I’m referring to, read the CircleCI blog post. It’s short, funny, and maddening all at once. We need to stop the never ending pursuit of the silver bullet and figure out a more sustainable way to build great services people find value in. Otherwise, I’m afraid we’re spending too much of our time porting or, worse, figuring out how to port existing stuff that works well onto something that is now deemed “better”.

Cloning VirtualBox images (or how I save hours a day when testing software)

First things first. If you’re testing software and you’re not using some sort of virtualization solution, stop reading this and go install one. My product of choice is VirtualBox. It’s free (as in no cost and most of it is open source), user friendly, runs on an Ubuntu host computer and I’m familiar with it.

Lately I’ve been doing a lot of testing of the Ubuntu One desktop software and I need to be able to quickly get various versions of Ubuntu up and running. Below I outline how I do that on an Ubuntu host computer. My steps assume that you’re familiar with VirtualBox enough that you know how to setup a virtual machine (VM) already.

Create a master image

The master image is the one we’ll use to clone test images off of. By doing this we can worry about keeping our master image up-to-date and configured the way we need it and then simply clone that image when we have to test.

  1. Create a new VM in VirtualBox and install the OS  (see Lifehacker’s guide if you’re not sure how to do this)
  2. After restarting the VM when the install is done, install all the latest updates on the master image and restart
  3. Install the VirtualBox Guest Additions (allows nice integration with the host computer)
  4. Shutdown the master image

Periodically you’ll want to make sure your master image has all the latest updates, so just boot it up, install the updates and then shut it down.

Clone the master image

Now we’re ready to start testing some software. Instead of using the master image we created above, we’re going to clone that image. This should take less than 5 minutes start to finish.

  1. In a terminal session do the following:
  2. In VirtualBox, create a new VM by clicking the New button
  3. Go through each screen selecting the appropriate values and clicking the Next button until you get to the Virtual Hard Disk part
  4. Select the Use existing hard disk radio button
    VirtualBox Hard Disk Setup screenshot
  5. Click on the folder icon next to the pull down menu listing existing VDI files
  6. Click the Add button
    Add VirtualBox VDI screenshot
  7. Select the image you created (should be in ~/.VirtualBox/HardDisks) to add it to the list of available hard disks
  8. Click on the image you just added and then click the Select button
  9. Click the Next button
  10. Click the Finish button

You now have a brand new VM to use for testing. Once you test with this image and decide its usefulness is over you can delete the virtual disk image (VDI) file in ~/.VirtualBox/HardDisks, repeat step 1 above to create a new cloned image, and then edit your cloned VM in VirtualBox to use the new clone image. In other words, you don’t have to setup a new VM (steps 2-10) every time you want to use another VDI if you don’t want to.

Review The Tests, Then The Code

magnifying-glass

Last week I was talking with one of the developers at work and he was telling me about improvements his team was making in regards to (informal/desk check) code reviews. I told him that was great and then followed that up with a recommendation on how to improve even more.  Next time there is a code review, start with the tests. This will do two things: 1) Stress the importance of having tests 2) Make the review go quicker, as the tests will either be there and make the intent of the code clear or they won’t be there and the review is postponed until tests are in place.

I know I’m not the first to make this suggestion, but I felt it was worth repeating. Code reviews do have value, even when you pair program. At least, I think they do. Pair programming provides two sets of eyes on the code at the time of creation, but it’s surprising what someone who hasn’t seen the code before will find during a review.

Oh, and if reviewing the tests during a code review doesn’t make the intent of the code any clearer, then I suggest the team investigate behavior driven development (BDD). I’m becoming quite partial to the Cucumber framework for BDD, as I like it’s emphasis on user stories with acceptance tests/examples written in plain text and backed by executable steps.

The Customers Disappear Right Before Our Eyes

Tonight I was foolish enough to think my family and I could waltz into a restaurant on Valentine’s and get a table without much trouble. We were trying a restaurant we hadn’t been to before and I realized as soon as we pulled into the parking lot this probably wasn’t the best night to experiment. We went inside anyway to see how long the wait was. Much to my surprise, the hostess told us it was only about a 15-20 minute wait. I looked around the restaurant and, while it was full, there wasn’t a large number of people waiting on tables. We got on the waiting list and then proceeded to wait.

What I saw during the wait was kind of surprising. I noticed that the staff were diligently doing their jobs, quickly moving from one spot to another. I had to be careful not to step too far away from the waiting area or I was sure to bump into one of the busy bee workers. And while the staff was incredibly focused on the operations of the restaurant, I couldn’t help but notice they were almost oblivious to their customers, whether those customers be waiting for a table, eating their meal, or somewhere in between. I looked around the restaurant of about 40 tables and noticed that at any given moment there were numerous people looking to get the attention of their waitress. Customers coming in through the door were rarely greeted in a timely manner. Those of us waiting for a table were ignored completely, even as our original wait time came and went.

It dawned on me that this situation is not unlike what happens with some software development projects. We get in the groove of producing the software and never take time to make sure we’re satisfying the customer. Yes, those of us practicing agile have the advantage of delivering in short iterations which, at worst, won’t allow a project to go too long before the customer is back in the mix. But, even on agile projects, I’ve seen teams go through an entire iteration without giving much thought to the customer’s needs beyond the initial planning meeting. We think we understand exactly what the customer wants, can’t or simply don’t get continuous feedback from the customer during the iteration, and we develop the functionality. We become like the restaurant’s staff, who are so busy running the restaurant that they forget about the very people who really keep the restaurant running, the customers. Sure, the orders are taken, the food is getting out to people, drinks are refilled, new people are put on the waiting list, etc., but the customers aren’t satisfied, let alone happy. The software gets designed, written, documented, tested, etc. but the customers aren’t satisfied, let alone happy. Sound familiar?

Tonight was a good reminder that technical and operational excellence is critical to delivering a good customer experience, but if we lose sight of the customer in that process then we will fail miserably. We need to be diligent about keeping constant contact with the customer and focusing on the value we’re delivering. We don’t want our customers to walk away from us like my family and others did on the restaurant we attempted to eat dinner at tonight.

One Smart Cucumber

I was messing around with some Ruby stuff tonight and was reading up on the Cucumber project. Cucumber is a behavior driven development (BDD) testing tool.

Anyone who talks to me about user stories knows I’m kind of a stickler on the value statement.  I think I’ve found a kindred soul in Aslak Hellesoy, the creator of Cucumber. The following comes from the Cucumber documentation.

You should discuss the “In order to” part of the feature and pop the why stack max 5 times (ask why recursively) until you end up with one of the following business values:

  • Protect revenue
  • Increase revenue
  • Manage cost

If you’re about to implement a feature that doesn’t support one of those values, chances are you’re about to implement a non-valuable feature. Consider tossing it altogether or pushing it down in your backlog. Focus on implementing the MMFs (Minimum Marketable Features) that will yield the most value.

Here is an example taken from an IRC chat session in #cucumber:

I was impressed to find that section in the documentation. Not only do the docs detail basic usage but they go into the deeper discussion of how to apply agile principles using Cucumber. The balancing act of teaching principles and practices at the same time is no small feat. To find that in software documentation is pretty amazing.

P.S. One of my worst titles ever, I know. I think I was subconsciously inspired by the Hallmark Movie my wife had on a couple nights ago. I now realize I missed my calling as a greeting card writer.

Apache Shindig PHP Setup Problem Solved

First things first. Shindig is an Apache incubator project that serves as an OpenSocial Container and…I’m already falling asleep. Too boring. I’m way behind the curve on all things “social networking” but I have an idea for an app that may scratch an itch I have, and it just happens that some of this social networking stuff might be a good way to jump start the idea. So that means catching up on a couple of years (or more) of stuff that’s been going in the world of web 2.0, which includes OpenSocial. I wanted a local OpenSocial test sandbox and found Shindig to fit that bill.

Now, onto the problem I was running into and the solution. Unless you want the long boring details, I can save you the trouble and tell you to make sure you have mod_rewrite enabled in Apache.

If you haven’t dozed off by now then it probably means you’re running into issues with the Shindig PHP setup and would like some more details. I followed the directions on setting up the PHP Shindig server on my laptop running Kubuntu 08.04 here at home and kept getting a 404 error. I thought that was strange since I could hit other files with my browser that I put in that directory manually for testing but not the Shindig index.php page. Turns out the Shindig index.php page tries to be more “controller/servlet” like by sniffing the URI and passing it along to the appropriate handler. If it can’t find a match, then it gives a custom 404 error. I noticed the test URI (http://shindig/gadgets/ifr?url=http://www.labpixies.com/campaigns/todo/todo.xml) didn’t have any reference to index.php. Ah yes, the wonders of Apache mod_rewrite! And guess what I didn’t have enabled? Yep, Apache mod_rewrite. So, below is what I did to get things running on an Ubuntu setup from scratch (meaning no Apache 2, PHP 5 was installed or configured):

Stop laughing, I use the Pico text editor. I’m, as my son would say, “weak sauce”. I know. Anyway, here’s what I have in that virtual host file:

Once we have that in place, we need to edit our host file:

Append the following to the hosts file and save:

Time to restart Apache:

Now you should be able to go to your web browser of choice and run the demo/test app:

If all went well you should see something a little like this:

That’s The Java Way!

A quote from an email I saw come across this evening:

Otherwise, what’s the point? Pile up one xml on top of the other? 

I couldn’t resist.  A cheap shot at Java.  I’m just bitter after being exposed to one too many Java frameworks in the past that seemed to think that “you can never have too much XML” was a mantra to proudly live by.