Add Changelog generation script for GitHub API

Required details are documented inside RELEASE.md

Fixes to the original file:
- duplicate issues (pagination with links instead of page (Github doesn't like counting))
- escape markdown in issue title
- use formatting helper functions
- env support for user, token, project
- "support" category for everything which is not "bug" or "enhancement"
- ignore more labels ("feedback", "question")
- sort milestones by due date and version string

refs #5457

RELEASE.md

@@ -2,7 +2,9 @@
 
 Specify the release version.
 
+```
 VERSION=2.7.0
+```
 
 ## Issues
 
@@ -17,29 +19,47 @@ master branch which should be part of this release.
 
 Update the [.mailmap](.mailmap) and [AUTHORS](AUTHORS) files:
 
-    $ git checkout master
+```
-    $ git log --use-mailmap | grep ^Author: | cut -f2- -d' ' | sort | uniq > AUTHORS
+git checkout master
+git log --use-mailmap | grep ^Author: | cut -f2- -d' ' | sort | uniq > AUTHORS
+```
 
 ## Version
 
-Update the version number in the following file:
+Fetch the latest spec file from the [icinga-packaging](https://github.com/icinga/icinga-packaging)
-
+repository and verify that the latest version is set.
-* [icinga2.spec]: Version: (.*)
-
-Example:
 
+```
 wget -O icinga2.spec https://raw.githubusercontent.com/Icinga/icinga-packaging/rpm/snapshot/icinga2/icinga2.spec
 gsed -i "s/Version: .*/Version: $VERSION/g" icinga2.spec
+```
 
 ## Changelog
 
 Update the [CHANGELOG.md](CHANGELOG.md) file.
 
+Export these environment variables:
+
+```
+export ICINGA_GITHUB_AUTH_USERNAME='user'
+export ICINGA_GITHUB_AUTH_TOKEN='token'
+export ICINGA_GITHUB_PROJECT='icinga/icinga2'
+```
+
+Run the script which updates the [CHANGELOG.md](CHANGELOG.md) file.
+
+```
+./changelog.py
+git diff
+```
+
 ## Git Tag
 
 Commit these changes to the "master" branch:
 
-    $ git commit -v -a -m "Release version $VERSION"
+```
+git commit -v -a -m "Release version $VERSION"
+```
 
 For minor releases: Cherry-pick this commit into the "support" branch.
 
@@ -48,30 +68,40 @@ releases) or the "support" branch (for minor releases).
 
 GB:
 
-    $ git tag -u EE8E0720 -m "Version $VERSION" v$VERSION
+```
+git tag -u EE8E0720 -m "Version $VERSION" v$VERSION
+```
 
 MF:
 
-    $ git tag -u D14A1F16 -m "Version $VERSION" v$VERSION
+```
+git tag -u D14A1F16 -m "Version $VERSION" v$VERSION
+```
 
 Push the tag.
 
-    $ git push --tags
+```
+git push --tags
+```
 
 For major releases: Create a new "support" branch:
 
-    $ git checkout master
+```
-    $ git checkout -b support/2.7
+git checkout master
-    $ git push -u origin support/2.7
+git checkout -b support/2.7
+git push -u origin support/2.7
+```
 
 For minor releases: Push the support branch, cherry-pick the release commit
 into master and merge the support branch:
 
-    $ git push -u origin support/2.7
+```
-    $ git checkout master
+git push -u origin support/2.7
-    $ git cherry-pick support/2.7
+git checkout master
-    $ git merge --strategy=ours support/2.7
+git cherry-pick support/2.7
-    $ git push origin master
+git merge --strategy=ours support/2.7
+git push origin master
+```
 
 # External Dependencies
 
@@ -113,23 +143,41 @@ already has chocolatey installed. Pull/checkout the release.
 
 Create the nupkg package:
 
+```
 cpack
+```
 
 Install the created icinga2 package locally:
 
+```
 choco install icinga2 -version 2.7.0 -fdv "%cd%" -source "'%cd%;https://chocolatey.org/api/v2/'"
+```
 
 Upload the package to [chocolatey](https://chocolatey.org/packages/upload).
 
 ## Online Documentation
 
-SSH into the web box, navigate into `icinga2-latest/module/icinga2`
+Edit `config.yml` in the [icinga-docs-tools](https://github.com/Icinga/icinga-docs-tools) repository:
-and pull the current support branch.
+
+```
+git clone git@github.com:Icinga/icinga-docs-tools.git
+cd icinga-docs-tools/
+vim config.yml
+
+git commit -av -m "Update to Icinga 2 v$VERSION"
+git push
+```
+
+SSH into the web VM and build the docs:
+
+```
+cd /var/www/docs/icinga-docs/latest/
+/usr/bin/bundle exec ./build-docs.rb
+```
 
 ## Announcement
 
 * Create a new blog post on www.icinga.com/blog
-* Send announcement mail to icinga-announce@lists.icinga.org
 * Social media: [Twitter](https://twitter.com/icinga), [Facebook](https://www.facebook.com/icinga), [G+](https://plus.google.com/+icinga), [Xing](https://www.xing.com/communities/groups/icinga-da4b-1060043), [LinkedIn](https://www.linkedin.com/groups/Icinga-1921830/about)
 * Update IRC channel topic
 

changelog.py

@@ -0,0 +1,187 @@
+#!/usr/bin/env python
+# -*- coding:utf-8 -*-
+
+import requests
+import re
+import pickle
+import sys
+import os
+from datetime import datetime
+from collections import defaultdict
+
+try:
+    github_auth_username = os.environ['ICINGA_GITHUB_AUTH_USERNAME']
+except KeyError:
+    print "ERROR: Environment variable 'ICINGA_GITHUB_AUTH_USERNAME' is not set."
+    sys.exit(1)
+
+try:
+    github_auth_token = os.environ['ICINGA_GITHUB_AUTH_TOKEN']
+except:
+    print "ERROR: Environment variable 'ICINGA_GITHUB_AUTH_TOKEN' is not set."
+    sys.exit(1)
+
+try:
+    project_name = os.environ['ICINGA_GITHUB_PROJECT']
+except:
+    print "ERROR: Environment variable 'ICINGA_GITHUB_PROJECT' is not set."
+    sys.exit(1)
+
+changelog_file = "CHANGELOG.md" # TODO: config param
+debug = 1
+
+#################################
+## Helpers
+
+def write_changelog(line):
+    clfp.write(line + "\n")
+
+def log(level, msg):
+    if level <= debug:
+        print " " + msg
+
+def fetch_github_resources(uri, params = {}):
+    resources = []
+
+    url = 'https://api.github.com/repos/' + project_name + uri + "?per_page=100" # 100 is the maximum
+
+    while True:
+        log(2, "Requesting URL: " + url)
+        resp = requests.get(url, auth=(github_auth_username, github_auth_token), params=params)
+        try:
+            resp.raise_for_status()
+        except:
+            break
+
+        data = resp.json()
+
+        if len(data) == 0:
+            break
+
+        resources.extend(data)
+
+        # fetch the next page from headers, do not count pages
+        # http://engineering.hackerearth.com/2014/08/21/python-requests-module/
+        if "next" in resp.links:
+            url = resp.links['next']['url']
+            log(2, "Found next link for Github pagination: " + url)
+        else:
+            break # no link found, we are done
+            log(2, "No more pages to fetch, stop.")
+
+    return resources
+
+def issue_type(issue):
+    if "bug" in [label["name"] for label in issue["labels"]]:
+        return "Bug"
+    elif "enhancement" in [label["name"] for label in issue["labels"]]:
+        return "Enhancement"
+    else:
+        return "Support"
+
+def escape_markdown(text):
+    #tmp = text.replace('&', '&amp;').replace('<', '&lt;').replace('>', '&gt;')
+    tmp = text
+    tmp.replace('\\', '\\\\')
+
+    return re.sub("([<>*_()\[\]#])", r"\\\1", tmp)
+
+def format_labels(issue):
+    labels = filter(lambda label: label not in ["high", "low", "bug", "enhancement", "feedback", "question"], [label["name"] for label in issue["labels"]])
+
+    if len(labels):
+        return " (" + ", ".join(labels) + ")"
+    else:
+        return ""
+
+def format_title(title):
+    # Fix encoding
+    try:
+        issue_title = str(title.encode('ascii', 'ignore').encode('utf-8'))
+    except Error:
+        log(1, "Error: Cannot convert " + title + " to UTF-8")
+
+    # Remove dev.icinga.com tag
+    issue_title = re.sub('\[dev\.icinga\.com #\d+\] ', '', issue_title)
+
+    #log(1, "Issue title: " + issue_title + "Type: " + str(type(issue_title)))
+
+    return escape_markdown(issue_title)
+
+#################################
+## MAIN
+
+milestones = {}
+issues = defaultdict(lambda: defaultdict(list))
+
+log(1, "Fetching data from GitHub API for " + project_name)
+
+clfp = open(changelog_file, "w+")
+
+with open('tickets.pickle', 'wb') as fp:
+    pickle.dump(fetch_github_resources("/issues", { "state": "all" }), fp)
+
+with open('tickets.pickle', 'rb') as fp:
+    cached_issues = pickle.load(fp)
+
+for issue in cached_issues: #fetch_github_resources("/issues", { "state": "all" }):
+    milestone = issue["milestone"]
+
+    if not milestone:
+        continue
+
+    ms_title = milestone["title"]
+
+    if not re.match('^\d+\.\d+\.\d+$', ms_title):
+        continue
+
+    if ms_title.split(".")[0] != "2":
+        continue
+
+    milestones[ms_title] = milestone
+
+    ms_tickets = issues[ms_title][issue_type(issue)]
+    ms_tickets.append(issue)
+
+write_changelog("# Icinga 2.x CHANGELOG")
+write_changelog("")
+
+for milestone in sorted(milestones.values(), key=lambda ms: (ms["due_on"], ms["title"]), reverse=True):
+    if milestone["state"] != "closed":
+        continue
+
+    if milestone["due_on"] == None:
+        print "Milestone", milestone["title"], "does not have a due date."
+        sys.exit(1)
+
+    ms_due_on = datetime.strptime(milestone["due_on"], "%Y-%m-%dT%H:%M:%SZ")
+
+    write_changelog("## %s (%s)" % (milestone["title"], ms_due_on.strftime("%Y-%m-%d")))
+    write_changelog("")
+
+    ms_description = milestone["description"]
+    ms_description = re.sub('\r\n', '\n', ms_description)
+
+    if len(ms_description) > 0:
+        write_changelog("### Notes\n\n" + ms_description + "\n") # Don't escape anything, we take care on Github for valid Markdown
+
+    for category in ["Enhancement", "Bug", "Support"]:
+        try:
+            ms_issues = issues[milestone["title"]][category]
+        except KeyError:
+            continue
+
+        if len(ms_issues) == 0:
+            continue
+
+        write_changelog("### " + category)
+        write_changelog("")
+
+        for issue in ms_issues:
+            write_changelog("* [#" + str(issue["number"]) + "](https://github.com/" + project_name
+                + "/issues/" + str(issue["number"]) + ")" + format_labels(issue) + ": " + format_title(issue["title"]))
+
+        write_changelog("")
+
+clfp.close()
+log(1, "Finished writing " + changelog_file)