Grant Proposal: Revitalize blogs.perl.org
Sat, 18-Jul-2015 by
Makoto Nozaki
edit post
< is in need of replacement. Evozon would like to offer a customized instance of [PearlBee](http://pearlbee.org) in its place, with source freely available on [GitHub](https://github.com), and hosted on [Evozon](http://evozon.com) infrastructure.
## Benefits to the Perl Community
In the last few weeks, blogs.perl.org has suffered from a number of authentication issues, affecting many people including the authors of this document. The current web host, [MovableType](http://movabletype.org) has not addressed these issues, and placed the fate of blogs.perl.org in question.
While many solutions have been suggested, Evozon would like to offer its services. We would like to provide a blogging platform based on PearlBee customized to the existing blogs.perl.org's requirements. Evozon will donate hardware and developer time in exchange for hosting blogs.perl.org on our servers. The new [blogs.perl.org](https://metacpan.org/pod/blogs.perl.org) code will be a publicly available fork of PearlBee on GitHub, customized to match the existing MovableType routes, and using the existing MovableType URLs so that extant links and search engine content remains in contact.
Since the goal is for this to be a community project, we'd like to integrate with [Travis CI](https://travis-ci.org) so that interested parties can keep track of progress and add their own comments and suggestions in the form of pull requests.
Most of the development activity is expected to be around matching schemas, and ensuring that existing users, whether they're logging in with their own username and password or with their OpenID account, can log in smoothly and be able to post and respond to postings.
Evozon will provide UI and JavaScript design services in order to give blogs.perl.org an appropriate look-and-feel.
## Deliverables
- Host [blogs.perl.org](http://blogs.perl.org) source on a public [GitHub](http://GitHub.com) repository
- Integrate Travis CI with the blogs.perl.org repository for continuous (and visible) integration and testing
- Bring blogs.perl.org up to feature parity with existing MovableType.org server
- Migrate MovableType.org database to blogs.perl.org schema
- Test alpha.blogs.perl.org in-house with special attention paid to existing users' login
- Public alpha trial of alpha.blogs.perl.org again emphasizing existing users
- Roll comments and bug fixes into beta.blogs.perl.org, again emphasizing login
- Release final blogs.perl.org on Evozon infrastructure
- Monitor blogs.perl.org after release, respond to issues as they come up.
## Project Details
The new blogs.perl.org will be a [PearlBee](http://pearlbee.org) fork written on top of [Dancer 2](http://perldancer.org) and [DBIx::Class](https://metacpan.org/pod/DBIx::Class), with a [mySQL](http://mysql.org) database imported from the existing blogs.perl.org, and session management provided by [Memcached](http://memcached.org). Development and test deployment will initially be done on [VirtualBox](http://virtualbox.org) instances, possibly migrating to [Docker](http://docker.io) containers later.
Internally we're using [Trello](http://trello.com) for our projects. we already have a board for PearlBee, but we'll create a separate blogs.perl.org board and give Dave plus the TPF grant manager(s) access so they can follow along and ask questions of the developers.
Having the application publicly visible on [GitHub](https://github.com) will also let others help spot potential security holes and other attack vectors, which is of course important because this is a publicly visible application that accepts several different kinds of authentication, displays user-composed HTML and reads/writes a database, all of these being possible attack vectors.
We should have two instances of the application running behind a software loadbalancer, so that we can stop one instance at a time in order to perform maintenance without interrupting service. Since the application has such a low database write rate, deadlocks and race conditions shouldn't be a concern. Evozon has data centers in the US, UK and Singapore should the latency accessing the site from overseas become a concern.
Unit testing will be performed through [Travis CI](https://metacpan.org/pod/travisci.org), and automated UI tests through [Selenium](http://seleniumhq.org) as they become necessary, although the good ol' Mark I eyeball will suffice for most purposes.
We'll start by performing a code review of the existing fork to make sure it conforms to current Perl best practices, and is suitable for deployment. When that's complete the next step will be to migrate the existing database into the new blogs.perl.org schema. Dave has assured us that he can share the existing MovableType password hashing scheme, so that we can continue to support existing users without asking them to change login IDs or passwords.
Once real users can log in, we'll turn our attention to making sure that existing blog posts are displayed with a new look and feel courtesy of Evozon's design team, with careful attention to UTF-8 encoding, code samples, syntax highlighting and comments.
After we've made sure that users can view their blog postings as they were meant to be seen, we have to make sure that users can view their old blog posts in the new editor and be able to save the edited versions in the original format, which will save hassle when we perform the final migration. I'd like to preserve beta users' posts, and making sure the editor is backwards-compatible is one important step in that regard.
Now, regular users should be able to log in, view posts and comments, edit existing posts and comments, and create new posts and comments. Since that's pretty much what a blog site is expected to do, at this point we can open it up to other testers on a limited alpha trial, with the usual understanding that their posts and comments may not make it into the final version.
We'll take feedback and comments from alpha testers, keeping in mind that lots of blogs.perl.org users live in the US, yet the servers are (for the moment) here in Romania. Hopefully there won't be visible latency, but it's something to keep an eye on. The comments we'll collect will lead to a beta release which will be open to all users, again with the understanding that their postings may not make it into the final release.
Then it's time for assessment, and getting ready for final release. If all works out, we should be able to obtain the final live blogs.perl.org schema, load that onto our local mySQL servers, deploy from Travis CI to live, and perform a final smoketest before throwing the switch.
## Inch-stones
Of course, no battle plan ever survives contact with the enemy, but this should give an idea of how we want to proceed.
- Set up infrastructure
- blogs.perl.org fork
- Travis CI
- Trello board
- outside access to testing systems
- Migrate the database
- Add foreign keys to MovableType schema
- Import into DBIx::Class
- Load MovableType users and blog entries into blogs.perl.org database
- Mesh blog contents
- View MovableType content in blogs.perl.org template
- Read MovableType blog content into blogs.perl.org editor
- Fix HTML editor to support blogs.perl.org content
- Write blogs.perl.org content back into database
- OpenID, Google authentication
- Add OpenID/Google authentication to blogs.perl.org fork
- Test with existing and new accounts, especially creating new accounts
- External testing
- Now that users can view blog posts, log in, create and edit posts, and log out....
- Release alpha version
- And fix bugs as they arise, making sure to add tests to Travis where possible.
- Beta testing
- Open beta.blogs.perl.org for testing
- Watch site performance, especially load due to static content and lag to the USA where most users will be.
- Fix bugs as they arise, and of course add regression tests to Travis where possible.
- Cut off the beta after, say, 2 weeks.
- Release time
- Prepare rollback plan in case it's needed, probably as simple as pointing DNS back to the original blogs.perl.org but we'll discuss with Dave.
- Take a final database dump from (now old) blogs.perl.org
- Import onto the live machine
- Bring up the memcached server pool
- Bring up the first instance, making sure an instance can survive on its own without its twin.
- Smoketest the instance, make sure the front page can be viewed and at least the admin and two (non-Evozon, not Dave) users can log in and edit posts)
- Bring up the second live instance, kill the first live instance and make sure users can still edit posts
- Bring the first live instance back up
- Open the floodgates
- Fix whatever problems arise, if any. Hopefully none.
- Make certain monitoring is in place and live, let the application run and answer whatever questions people may have.
## Project Schedule
I'd love to release beta.blogs.perl.org in time for YAPC::EU, but that's too aggressive of a schedule, especially given that we're moving offices in August. The three main issues I can see here are the database migration, user authentication and finally making sure the old blogs.perl.org content is compatible with the new system.
We'll have to start off making sure that the existing PearlBee fork can support the existing blogs.perl.org content, namely user administration, privileges, and the various URL routes. This will probably take about a week of poking around in the various corners of the application because any major changes or additions would require database changes, and those should be done before I start writing a migration process.
Once that's done, the database migration will probably take 2 weeks, mostly to be able to pick apart the old schema.User authentication is always a concern, and as such we'll schedule 2 weeks for whatever issues we find. Finally, 2 weeks to impedance-match the new HTML editor with the existing MovableType content seems a bit of overkill but should allow us some slop should other bits go over schedule. The actual content looks like it's mostly groomed HTML, so it __should__ go smoothly, but we have to make sure that it displays in the new editor, and the new editor can save text.
Setting up the infrastructure here probably will take about a week at the most, and I can do that in parallel with checking over the application for missing features. Mostly it'll involve setting up VirtualBox instances, getting Amalia to create a new Trello board and give outside people access to it. I've (Jeffrey) been meaning to get a Travis CI account anyway, and will link that to blogs.perl.org.
Ignoring time out for YAPC and time for the office move, a tentative schedule looks like this:
- Administrative tasks, checking PearlBee for needed features: Aug 2-8
Most of the admin tasks will be handled by others, I just have to keep a checklist.
- Database migration - Aug 9-22
It really shouldn't take this long, but I'm factoring in searching for and adding foreign keys to a database that wasn't equipped for them, always fun.
- User authentication - Aug 23-Sep 12
Lots of slop there, Dancer2 plugins should be able to handle most of the tasks. I'm factoring in time for getting keys from various parties and integrating the logos on the new front page, we'll have to have the designers come up with an appropriate UI.
- Matching blog content to the new blogs.perl.org editor - Sep 13-26
Probably some slop here as well, because MovableType stores things in HTML and that should be easy to integrate.
- Public beta - Sep 27-Oct 3
I'm not envisioning any major fallout from this, but then again Perl hackers can be a persnickety bunch. Hopefully they'll do their best to break our HTML rendering engine, and we can keep an eye out for that.
- Release - Oct 4-Oct 17
The fun part of any schedule. I've expanded it to 2 weeks because we'll have to coordinate with Dave Cross in order to transfer the old domain and solve any authentication and security issues that arise from moving shared secrets between IP addresses and to new domain owners.
## Completeness Criteria
- blogs.perl.org on GitHub and integrated with Travis
- blogs.perl.org feature complete
- Live schema imports via DBIx::Class into blogs.perl.org with no errors or constraint violations
- Blog text, code snippets, quoting and comments display in clean UTF-8
- Admin and test users can log in to their account through direct access
- Beta users can log in through OpenID and whatever other IDs we support at rollout
- Live users can view content, log in with their existing ID, create and comment on content
## Bio
Jeffrey Goff is a long-time member of the Perl community, CPAN author, speaker at Perl conventions worldwide, active on IRC for an embarrassingly long time, one of the original Perl6 release managers and an active Perl6 core contributor. Before moving to Evozon.com and taking over buitinperl.com, he worked on release teams for the website builder [MoonFruit](https://moonfruit.com) and architected a Hadoop pipeline for [TagMan](https://tagman.com).
Amalia is has been involved in the Perl community since starting the local Perl Mongers group, [Cluj.pm](http://cluj.pm), as an initiative of the Perl Department at [Evozon](https://www.evozon.com). She has been organizing the meetups of the Cluj community since March 2012, and also attending worldwide Perl events, like YAPC::Europe, London Perl Workshop, or FOSDEM, either as a regular attendee or volunteer.
Her involvement in the Perl Community has been officially recognised, in December 2014 being the proud recipient of a White Camel Award.
In addition, she is also involved in the development and growth of builtinperl.com.
]]>
Comments (17)
An improved blogs.perl.org that still runs on Perl seems like a good thing. Movable Type is proprietary and seems to be a bit of a dead end. This at least would be open.
That said, a custom one-off fork of an existing project seems a little risky too. If there's no hope of anyone else using this I worry that it just won't receive that much attention (in the form of patches).
The current blogs.perl.org team is all in favour of this project.
I vote for supporting this project.
"The current web host, MovableType has not addressed these issues, and placed the fate of blogs.perl.org in question."
Some slight confusion here that I should probably clear up.
Movable Type is not the current web host. The site is hosted on a dedicated server that Aaron and I rent for the purpose.
When we set the site up in 2009, Six Apart (the owners of MT) donated a professional MT licence and a lot of consultancy time to get the site running. We are grateful for these donations.
The current problems are caused by us being unable to upgrade our version of MT to newer versions - partly because there is no clear upgrade path from the professional version that we are using and partly because we don't know if the new owners of MT (a Japanese company who also bought the name Six Apart) would be in a position to honour our existing licence.
So whilst we do think that the best approach for blogs.perl.org is to move to a new platform (hence this grant proposal) none of the blame for this existing problems should be attributed to either the hosting company or Six Apart.
With regards to Dave's point about the license, I just got the following from Six Apart:
> We are happy to offer the latest version of Movable Type for blogs.perl.org
...
> On the OpenID Connect issue, we will support in the next version or later"
(Note: I am not in the position to say which direction is better and I don't have intention to influence the decision in one way or another. It's just for your information)
A better option may be to do a "call for proposals" so that various perl friendly hosting companies can solicit offers. A company may be interested in having their logo / banner placed strategically, especially if they are trading a common hosting plan for the privilege.
I dont know which software is best (MT, WebGui, others... bespoke?), but using perl seems like an obvious requirement. If an MT license has been purchased it also seems to make sense to find out if this license is still current.
A licence hasn't been purchased. A licence has been donated. And it sounds like Six Apart are happy to donate a similar licence for the latest version.
I don't know enough about the merits of PearlBee or Movable Type to comment on them, but as a non-voting TPF grant manager I'm impressed with the planning involved in this proposal. I would happily manage a grant like this.
Many people have frustrations with the current blogs.perl.org platform. I would like to see TPF support work to overcome these frustrations.
As the main author of the proposal, I apologize to MovableType and SixApart, and never meant to slight them. My discussion with the members of the blogs.perl.org maintenance crew were all focussed on "Where do we go from here?", not worrying about the existing maintenance issues. The existing bugs were never discussed, and I made some incorrect assumptions about why they hadn't been addressed based on other conversations.
PearlBee is a bog-standard Dancer application, so I expect that most of the development time will be shoehorning the existing MovableType data into the PearlBee schema and getting the layout to look how our UI designers want. Any code patches for blogs.perl.org will go back into the PearlBee mainline, and I'll also set about packaging it for CPAN so that it can go into wider dissemination within the community.
Evozon is offering hosting, design, development and maintenance time gratis to the Perl community, and in return we're offering to be as transparent as possible in our development efforts. My abject apologies again to MovableType and SixApart, what I said earlier was based on invalid assumptions.
Dave, we will donate a license of Movabel Type latest version with pleasure.
The current Six Apart is the former "Six Apart Japan" team. Now we own Movable Type and Six Apart treadmark (but no TypePad).
yes++
I really support this grant.
Reasons:
1) blogs.perl.org really needs some love.
2) PearlBee is a good blogging platform. I played with it a bit, it seems to be easily extensible. (Though I'm not at all aware of advantages of Movable Type over PearlBee).
3) This proposal looks very well-planned. Jeff and Amalia are smart & hard-working peeps. I'm very confident about them.
I'm really looking forward to use painless blogs.perl.org!
blogs.perl.org should be ported to Angerwhale.
I personally have no stake in what gets used to develop the site, PearlBee was simply chosen because that's what Evozon developed in-house; before I got involved, PearlBee's author was in upgrade discussions with the site admins. Looking at Angerwhale the last update was made roughly 6 years ago, and paradoxically Angerwhale-NG's last update was 7 years ago. I haven't looked into the code, but if it can be installed and has features that PearlBee is lacking we'll certainly consider it. While I imagine Amalia would want the final word on what actually gets used, I suspect she'd agree that if using Angerwhale saves time, we could use that.
I'm missing a few stages here. It's nowhere defined what feature-set the site needs. MT has lots of features that we don't use, and probably a few that we want that it doesn't have (ease of use being one of them, IMO).
Without that, one can hardly decide if this is a the best approach in theory, let alone in practice.
Also, simple blogs are generally considered easy to implement. I have the impression that, much like MT, Pearlbee was written with a very different feature set in mind and as such it's quite well possible that either writing a custom tool from scratch, or writing a more extensible tool is a better approach.
I may be talking out of my arse here, I'm not in the website-building business and have only briefly looked at the code.
I very much agree with the goal of having a better blogs.perl.org, but I don't think this plan of attack is making sense.
I should explain that I basically had four days to write and vet the proposal in between other work commitments, so you probably had more time to review the PearlBee code than I did. I'm sure PearlBee is missing features that we need to support from the old platform. Given how long b.p.org has been around I'm sure every feature it has *someone* is using. And getting a tick-list of features wasn't as much of a concern as getting a proposal out the door.
And I should point out that by the time I'd been asked to join the project, PearlBee had already been established as the codebase by everyone else. It's Dancer, designed in-house and had people willing to hack in new features for me as needs be. As I've said before I don't care what platform gets used as long as it gets the job done. Going to a Rails implementation might be going a bit far though :-) There probably are other alternatives out there but I was asked to write the proposal assuming PearlBee was the platform.
When writing the document I was focusing mostly on how to support existing users seamlessly and how best to migrate the old database. I'd had a few hours to look over PearlBee at most, before I had to start writing in earnest. Feature parity wasn't as much of a concern as being able to merge the old and new schemas without a headache, and making sure we could use the existing MT authentication code.
I don't know how much time I'll have at YAPC::EU to talk as I don't yet know when I'm leaving, but I'll suggest a b.p.org BOF session, regardless of what happens with the grant. That way I can address these concerns in person and we can discuss what will ultimately happen.
In the description you mention forking PearlBee in order to support MovableType's URLs, and possibly other reasons like better syntax highlighting. I think without looking at any source code or knowing how either MovableType or PearlBee are put together it would make more sense to just contribute these features directly back to PearlBee. Perhaps add an option to its configuration file to pick native, PearlBee, URLs or MovableType-like URLs. I don't know if PearlBee's design is flexible enough to easily support adding such an option to it, but if it is it is much better than forking PearlBee to add only one feature to it. Also, this fork could damage PearlBee itself, because people could wind up using this fork instead of PearlBee itself, because they want to replace their MovableType blog with PearlBee, which PearlBee itself does not support.
Thanks for taking this on.
