Week 3..orrr 4* on Reproducible Builds

Today I am in Cape Town for DebCamp+Conf!

Yesterday I traveled here from Boston.

The previous four days (Mon-Thurs) I converted (almost entirely) the bash script that create the package set pages of test.reproducible-builds.org to python. This involved writing more mustache templates, discovering old templates (read: strings of html) in multiple places and moving the package set definition to keep it DRY.

These changes are not yet deployed – here are things I’d like to fiish first:

1. Create a mustache template for the main left side navigation (the comment at the beginning of the bash function that presently builds this html: “this is really quite uncomprehensible and should be killed”). This navigation is used many places, including the front page. This is the only part of the package page html left to convert.

2. Use the mustache templates in the bash scripts where appropriate (there is a bash interpreter of mustache templates!). This will DRY up the code some more πŸ™‚

3. Add some navigation improvements to the package set pages. For example, from the package set page that shows you a summary of packages statuses on testing+amd64, you cannot navigate to the summary of the same package set on unstable+amd64.

4. Improve the main left hand navigation in point (2) above. Similar to the package page improvements, I’ll add hover text and headers and move things around.

*changing the number of the weeks to match other outreachy participants

Week 2 on Reproducible Builds

Here is a small update for last week’s work:

1. Templates

tests.reproducible-builds.org now has a templating system, instead of writing HTML in python files πŸ™‚ We are using mustache with pystache. So far, only the html for the package pages have been converted.

2. Navigation improvements on package pages

While converting the package pages to using mustache I also rearranged the navigation bar, adding sections and hover text. Navigation improvements inspired by Gerts Wollney’s email requests.Β  Please check out the changes and provide feedback: tests.reproducible-builds.org/<your-favorite-package>

3. Began converting script from bash to python

Before converting the rest of the site to mustache, I spent a day digging into the last unconverted bash script (all other html producing scripts are python). I’ve never written or read that much bash, yet another learning opportunity – I have to admit it’s a bit less intuitive than python πŸ˜‰

Up next:

Finish converting package set script, create more mustache templates, more site improvements! Then fly to South Africa.

Week 1 on Reproducible Builds

In this post I’m reviewing what I’ve done the last 6 days of Outreachy-funded reproducible builds work, outline what I plan to do the next two weeks, and speculate on long term goals. For those of you involved in the Debian reproducible builds project, please provide feedback about future plans and work!

Week One review

One week of Outreachy completed! What have I done?

  • Reproduced the reproducible builds tests website locally
  • Added information to the INSTALL file about reproducing the tests website (viewable here)
  • Checked in changes that broke nearly every link to the tests website
  • Fixed mosts of the broken links by adding redirects. (Please let me know if you find any!)

The change that broke everything was the addition of a directory: tests.reproducible-builds.org/debian

The directory was added to contain all Debian-specific pages, in line with the other project’s reproducible builds status pages: arch linux, fedora, coreboot, etcs. Previously, all Debian pages we simply served directly out of the DocumentRoot. To fix all the broken things, I’m pretty sure I had to find, inspect, and addΒ  /debian or change global variables within every file pointer in the entire tests website. Sometime tedious, but chasing down bugs and complaints was mostly fun πŸ™‚

I also learned (everything I now know) about Apache websites, redirects, the website/navigation/directory structure of tests.reproducible-builds.org, and the roles of many of the reproducible scripts in jenkins.debian.net/bin.

Week Two plan

What will or should I do next?

In the short term, over the next two weeks, I hope to make useful improvements to the tests website and backend while continuing to get up to speed (as well as learn Python).

  • Improve navigation on tests.reproducible-builds.org/debian
    • fix highlighting in the nav bar on package pages
    • address navigation bar re-organizing requests like this one
    • add documentation/hovertext for links
  • Create a front page for the test reproducible builds site (update: probably not do this yet, low priority, already “too many front pages” for reproducible builds)
  • Convert bin/reproducible_html_pkg_sets.sh to python

Have other thoughts about minor improvements to tests.reproducible-builds.org? Please let me know! The above list is not internally prioritized, feel free to ask for things to be bubbled up.

Longer-term goals

My long term summer goal is to make the Debian test code more easily extensible to show the reproducible results from other projects. This will lower the barrier for new projects to keep track of the reproducibility of their code, for great good.

This starts with the reproducible.db database, which presently only tracks reproducible testing results for the Debian project.Β  The reproducible builds project’s needs have outgrown the original SQLight database, so this redesigning includes a migration to Postgre. Goals of the redesign include ease of querying/comparing packages across distributions, as well as generalization to include results from projects other than Debian. I’ll start on this work in two weeks, when I get to DebCamp! πŸ™‚

Redesigning the database will also lead to updating the python script which use that data to produce the Debian tests website. Other project scripts (like Fedora, RedHat and Coreboot) can then be updated to track results in the database as well, instead simply directly producing their own test websites.

update: as an intermediate step — before redesigned the reproducible.db database to handle multiple projects — h01ger recommended I help the FreeBSD project recorded tests to a FreeBSD specific database.