So the realization that I had due to Tyler. He made a comment about how he doesn’t really write big websites or all in one bundles but rather uses and writes smaller services that talk to each other.
I had been thinking about how I wanted to integrate with DailyMile but couldn’t decide on how I wanted to pull down the data. After hearing how Tyler works I decided I’d just write a simple daemon in Lisp that could pull down my data and store it in the PostgreSQL database I planned on using. This made it far simpler to (start) writing both sides.
I am currently spending time on the daemon and how/what I want to store the entries in the database (so I don’t have to request them every time) the benefits of being able to focus solely on the communication between my daemon and the DailyMile service have made it so I am not solving problems like how to display the information or how to aggregate it or the features I’ll have in the web side of the app.
Once I have it pulling down and saving the basics of the entries consistently I can start into the web site side of things, which since it only has to read from the database not pull down data from a foreign server, it will simplify the site and I can focus on the features and processing the data.
Sure there are still plenty of problems to solve in this space and with what I want to do, but separation of concerns like this results in easier to maintain, extend and reuse services.
Those of you who read my blog will notice the new widget on the side. That is a widget from DailyMile as I have several goals right now. Ride a century (100 miles) on my bike, jog for 30 minutes straight without slowing to a walk or stopping, and finally to swim a mile straight without resting.
Now you, most of you who read this are going ‘wtf this is a tech blog’ ’tis the truth, this is a tech blog. First I wanted to give a shout out to the folks at DailyMile. Their site is pretty awesome, works (so far) without any noticeable bugs or flaws, and they are nice enough to provide a RESTful API that returns JSON (yay no XML-RPC). Heck, I’ve even contacted them about donating or paying for some pro-mode if they decide to add one. I know they offer shirts but those have an overhead cost to them and they aren’t really my style.
More on the API, it is a pretty decent setup, uses OAuth2, returns JSON, and is pretty intuitive. I am going to be building out some software of my own that uses it to track smaller goals with end dates and such which their ‘challenge’ system doesn’t really support in a robust way. The details of that along with a recent realization thanks to a fellow Portland Nerd will be coming in the next post. For now, I just wanted to show my love for this site that is already making my life a little better.
Tonight I set out for two goals, to start an app for tracking my progress in preparation for a Century Ride in the next few months, and to organize my fabfile.py to be more generic. I’ve started the app, though all it does is let you log into the admin section of the site (using django-admin) which is some success. But the real technical success tonight is my fabfile and how far it has come.
I had a monolithic fabfile that I have been copying between projects and just search and replacing in, it could (potentially) do a full install of an app, at a push of a button. Really those steps are things that are done (for me at least) once a blue moon. The more valuable part is push button releases. So I reworked it to only include the things I need for development and releases this is the result: Century fabfile.py
The lower the bar, the better for releasing new version of a site I am working on. In the past I had found myself editing python on the live server, then committing back to github as things work. This is poor practice for many reasons… Now, I can finish a feature, close a bug, or clean up code and easily deploy it to the live server with a simple `git push && fab deploy` The only reason the git push isn’t part of the deploy is that sometimes I unpushed code that is not ready for live, and I’d like to be able to deploy without having to worry about that.
I highly recommend that you guys check out Fabric, it is really simple to use and will make your life better in many ways.
So, here at my company we’ve been slowly adopting a more agile-like process. Which started with adding lots of tests and using CI to run the full test suite. As well as continual refactoring of code (once it is tested) to both make it fit the new requirements and to make it simpler. We have several release windows per week and attempt to utilize at least 1 or 2 of those, so the customer is getting our improvements as we complete them rather than in giant releases every few months.
And as we are integrating more and more agile practices into our work flow, we are hitting a problem that I have seen at a couple companies and have yet to see much that tackles this issue. Now this is nothing agile specific… it seems to happen with any sufficiently large application.
The problem is we have a backlog that is growing faster than we close tickets. Our first solution to this problem was to start having what we call ‘Bug Fridays’ and that has helped.
A Bug Friday is where we (as a company) can work on low hanging fruit, regardless of what our current projects are and how much value the low hanging fruit is worth. Since we are getting a lot of tickets done, the value adds up quickly, it makes both developers happy (because they closed a lot of bugs, especially old ones) and the client happy (because while it isn’t their highest priority stuff, often this are mild annoyances that they deal with every day)
This has had a positive effect and has even got us with a net decrease in bugs since we started them a month ago. The first one was most effective, the second one was less effective, and I am betting our third one (tomorrow) will be less effective than the second. We are doing them every two weeks and the amount of low hanging fruit in the backlog is decreasing each time. So long term I am not sure how valuable it is, it may need to become a once a month thing instead of every two weeks to allow for more low hanging fruit to accumulate.
From there, I am not sure what to do to stave off this growth of open tickets vs closed. I’d love to hear some suggestions and hear what other companies and projects are doing to attempt to win this battle.