Three is the number of those who do holy work;
Two is the number of those who do lover's work;
One is the number of those who do perfect evil
Or perfect good.Clive Barker
This is a story about pushing boundaries with Atlassian tools, putting more power into developers' hands (whether for good or evil...), and learning along the way.
I work for TechTime (an Atlassian Platinum Solution Partner based in New Zealand) as an Atlassian App Developer. However, my background is actually in Statistics. So when a task came to the team to create and share a monthly report based on analysis of raw Atlassian Marketplace sales data, it naturally fell onto my plate.
The problem was something like this:
Atlassian's Server licensing is tricky, sometimes people renew early, upgrade, co-term, migrate to Data Center or buy a whole new license instead of renewing and sometimes they may just renew 3 years after the initial 12 months of support and maintenance had ended.
We knew when people renewed their Server app licenses, but we didn't really know how many should have renewed but didn't.
We didn't know if the number of renewals we're getting is better or worse than usual, and we had no idea whether the economic impacts of COVID-19 were affecting our sales.
We wanted to understand our current renewals better and be alerted as soon as there are any problems or significant drops in renewals.
The easy option was to throw a bunch of scripts over the wall for your friendly neighbourhood operations team to worry about deploying, running, and monitoring. Instead, we chose to take advantage of the tools that are close to any developer's heart: build servers (Bamboo), Git (Bitbucket), and REST APIs (in this case using Opsgenie, Confluence, and the Atlassian Marketplace)
Unique features and integrations of many Atlassian tools come together in this use case, which has made it really easy for our Dev team to create scheduled tasks with great monitoring, logging, notifications, reliability, and security out of the box. No Ops engineers were harmed in the course of this project.
Here is the basic set up:
- A Bitbucket repository which contains several Python scripts for downloading data, analysing it and creating a report
- A Bamboo plan configured with a 'Schedule Trigger' to run at the start of every month
- The Bamboo plan checks out the repository and runs the scripts when the schedule is triggered
- The script downloads data from Atlassian's Marketplace API and calculates a bunch of statistics based on the sales for our apps. This includes calculating the actual monthly renewal rate: what percentage of customers who should have renewed this month, actually did so?
- If the actual monthly renewal rate of our apps falls below a threshold, Opsgenie sends a notification to a Slack channel, to ensure the right people see the result.
- Using a Bamboo Deployment Project, the generated HTML report is uploaded to our local Confluence as an attachment
Since all of this is running as a Bamboo task, we get a lot of nice bonuses, for no extra effort. 
Bamboo Logging
Now, of course, the script didn't run perfectly the first time around. Since I had set up Bamboo to send email notifications if the build had failed, I was able to quickly determine my scripts had failed miserably. Luckily, thanks to the logging that's part of every Bamboo build, I was able to easily see what went wrong when the script failed and fix it really quickly. To deploy the fix, all I had to do was commit to the repository, and trigger the build again.
Isolated Environment
Anyone who has to run a lot of scripts from one server will tell you, managing dependencies can be a nightmare. Since our scheduled tasks are running on a Bamboo agent, we have all of the tools available to create an isolated environment (just like you would need in any complicated software project). In this case, I set up my Python scripts to run inside a Docker Environment, which was as easy as selecting a checkbox in the Bamboo plan and selecting the docker image to use.
Access Control, Audit Logging, and Security
Since this is all set up as a Bamboo plan, we can set fine-grained permissions on who can view the results of the task, who can edit the plan, and who can create and edit the scheduled tasks. Of course, to set this up in a traditional Ops environment is possible, but by using Bamboo, everything is there by default.
Bamboo also gives secure options for storing details like the Opsgenie API key or any credentials that would be needed for connection to remote systems. Again, this is possible to achieve in a traditional setup, but using Bamboo means we get it without having to even try.
Alerts and Notifications - via Opsgenie
A monthly report with colourful charts and carefully crafted statistics is great, but sometimes we just need a traffic light - "are we OK"? Since we use Opsgenie for handling alerting relating to our apps anyway, we decided to see if Opsgenie could alert the team when any of the metrics calculated in the report show any significant drop.
Turns out, it's surprisingly simple. We ended up creating an "API Integration" in Opsgenie, which alerts based on values sent in the rest request.
Opsgenie doesn't do number based matching out of the box, so we configured a Regex matcher, to match if the renewal rate was below a certain threshold.
So all we need to do is send a request to Opsgenie once the analysis is complete, and Opsgenie can decide who on the team to alert depending on how bad (or not) the result is. In our case, if our renewal rate ever drops too low, we will be notified in a Slack channel, along with a link to the full report.
Deployment - to Confluence
Since the report is meant for others to see, we decided the monthly report should live in a place where the rest of our knowledge base is: Confluence. Again setting this up was straightforward, we created a page to house the reports, using Confluence's built-in Attachment Macro and upload the report as an attachment every time the build runs (we use a Bamboo secret variable to store the authentication details). So interested people can simply watch the page, and get an email every time a new report is uploaded.
Why not just add to Crontab?
As a developer, using a Crontab for any serious task usually involves roping in an unsuspecting Ops team member to help you safely deploy and manage your scripts.
If not deployed and maintained in an organised manner, you can end up with arcane and obscure scripts spread across several different servers, modifiable by anyone, and with no insight into why they were placed there in the first place.
Summary:
So far the system has been working well, so well that the Ops team has started adding some scheduled tasks in the same way! We have also done a few optimisations in Bamboo, such as running builds for these scheduled tasks on smaller, more cost-effective Elastic agents.
According to Ed (our CTO), using Bamboo to run scheduled tasks is a bit like using a cannon to shoot at sparrows (apparently this is a Russian expression?!), but so far the results have been very positive.
One thing that has stood out in the process is how the easy integrations with many Atlassian products have made it simple and seamless to create quite a robust process, linking several complementary products into a single unified solution.
What do you think? Perfect good or perfect evil? Too much for a simple scheduled task? How does your Dev team manage regular jobs such as the creation and distribution of monthly reports and jobs?