Grant Proposal: Perl::Lint - Yet Another Static Analyzer for Perl5

13 Comments

We have received the following grant application. Please leave feedback in the comments field by March 22nd, 2014.

Perl::Lint - Yet Another Static Analyzer for Perl5

  • Name:

    Taiki Kawakami (@moznion)

  • Email:

    undisclosed

  • Amount Requested:

    $800

Synopsis

This project aims to create a fast and flexible static analyzer for Perl5 that has compatibility with Perl::Critic, with the goal to be light and fast enough to allow near real-time check of Perl source code while you code (e.g. hook it in your editor)

Why do I fork Perl::Critic?

Currently Perl::Critic uses PPI to analyze perl source code, which is relatively slow. The processing time can be greatly shortened by about 30 times the current value by replacing PPI with Compiler::Lexer and Compiler::Parser, which are new tools created by Masaaki Goshima who authored PerlMotion.

Using these modules are the main ingredients to the speed up, but in my previous attempts introduce these changes to Perl::Critic, it has proven difficult to do this without changing the current codebase significantly thereby causing significant trouble for the current user base. It would especially affect the already-existing plugins for Perl::Critic.

Rather than breaking existing functionality, I propose to build a new tool that is feature-wise compatible to Perl::Critic, but with much faster processing speed.

Benefits to the Perl Community

While Perl::Critic is an extremely useful tool, we have seen many colleagues avoiding its use because of its slowness. Improving the execution speed of Perl::Critic would lower the barrier to use in your day-to-day hacking activities, thus also leading to a better quality code, and improved development efficiency. With a fast enough Perl::Critic, one might even be able to embedded in an editor for seamless and continuous checking as you write.

Deliverables

The new module which is compatible with Perl::Critic. And this module is much faster than Perl::Critic because it uses Compiler::Lexer and Compiler::Parser instead of PPI.

Project Details

This project aims to create a MUCH faster version of Perl::Critic, using Compiler::Lexer and Compiler::Parser

Compiler::Lexer and Compiler::Parser are tokenizer and parser for Perl5 written in C++, which offers unparalleled speed compare to pure-perl competitors used in PPI. They are used in projects such as PerlMotion which transpiles Perl5 code to Objective-C.

A real life example replacing PPI with Compiler::Lexer can be found in Perl::MinimumVersion and Perl::MinimumVersion::Fast . The former is implemented using PPI, and the latter uses Compiler::Lexer. A simple benchmark shows that Perl::MinimumVersion::Fast is about 33 times faster than Perl::MinimumVersion (https://metacpan.org/pod/Perl::MinimumVersion::Fast#BENCHMARK). Applying this to Perl::Critic, one can expect at least an order of magnitude faster processing.

Therefore goal of this project is to create this much faster replacement while still keeping compatibility with Perl::Critic.

Inch-stones

  • Analyze functions of Perl::Critic (half to one day)

    Extract items that Perl::Critic currently checks

  • Implement tokenizer (two or three weeks)

    Parse source code written in perl and divide to tokens by Compiler::Lexer and Compiler::Parser.

  • Implement code analyzer (two or three weeks)

    Analyze and evaluate tokens by checking with inspection rules

  • Implement a system to define inspection rules (one or half week)

    Implement a system to define inspection rules on this phase, as well as a basic set of rules that are frequently used (mostly ported from original Perl::Critic rules)

Project Schedule

Period: within two months When can I start: May 2014

Completeness Criteria

Pass all of tests for Perl::Critic, benchmark against Perl::Critic, and release the module to CPAN.

Bio

I am a software engineer and graduate student (majoring in high-performance computing and software engineering) in Japan. I'm experienced in systems development for about five years, and I have worked with perl for about 2 years. I have contributed to a few open source projects (mostly involving perl) and have released a few CPAN modules. Please refer to the following web sites for details:

https://github.com/moznion https://metacpan.org/author/MOZNION

I am also a member of Compiler::Lexer development team, so I know the inner implementation of these modules which are crucial for this project. I believe I can apply those skills to this project.

ACKNOWLEDGEMENT

While the original proposal was written by Taiki Kawakami, Daisuke Maki was responsible for editing this proposal's English.

13 Comments

I endorse this grant.
We should really go on with Compiler::Lexer and Compiler::Parser, and a linter is the easiest way to follow.

The interpreter is a long shot, the parser is not re-entrant yet. (eval)

The proposed benefits are clear and compelling and the example of Perl::MinimumVersion::Fast gives me confidence that this project can succeed. I fully support this grant.

(If I may suggest a follow-on idea, please consider doing the work in a way that would allow a faster and more powerful Perl::Tidy as well.)

This looks very interesting. To me this is clearly the best of the submitted proposals.

(My suggestion for a follow-on would be a faster Perl::PrereqScanner, but that's another discussion).

A framework for reliable and _fast_ tokenizing/parsing of Perl code (without simultaneously executing it) would enable the creation of a whole new generation of tools.

Also the amount of money requested is (surprisingly) low for the workload.

If it turns out that either time or money won´t suffice to reach the original goals I´d suggest to focus on the low level parts: feature completeness and API/ABI of the C++ library, not Perl::Critic compatibility.

This looks useful. I'd love to be able to speed up my code that uses PPI, even if it requires porting to some new parser/DOM. This seems like a good place to explore.

I agree this is an excellent proposal. More work on these kinds of tools is healthy for the Perl ecosystem.

This is an excellent proposal. The speedups demonstrated by Perl::MinimumVersion::Fast show that applying the same approach to Perl::Critic is likely to work well. I hope that this turns out to be the case.

As everyone else is suggesting a follow-up, may I too? The CPAN testers results for Compiler::Parser suggest that currently it only builds cleanly on a few platforms. Maybe addressing portability would be a good candidate for a future grant proposal, if someone feels able to. But to be clear, please don't see those test results as a blocker for accepting or working on this grant. Compiler::Parser already passes 100% on enough platforms that things built atop it will benefit a significant part of the Perl userbase.

+1
Very good proposal. That's exactly for this kind of projects that grants should be targeted. I expect it will have wide benefits for the whole Perl 5 ecosystem.

Thank you for a lot of feedbacks.

By the way, I made a lightweight prereq scanner by using Compiler::Lexer to ensure the solution which is proposed (inspired by Mr.Leon's post).

https://metacpan.org/release/Perl-PrereqScanner-Lite
https://github.com/moznion/Perl-PrereqScanner-Lite

The new one is faster about 30 times than Perl::PrereqScanner.
Therefore Perl::Lint also can realize acceleration, I think.

I would definitely want this. I've seen goccy's talk at YAPC::Asia 2013 and it was amazing; there's little doubt that static analysis using these tools is very nice.

+1

I would love a faster Perl::Critic.

Leave a comment

About TPF

The Perl Foundation - supporting the Perl community since 2000. Find out more at www.perlfoundation.org.

Recent Comments

  • Flimm: I would love a faster Perl::Critic. read more
  • Burak Gürsoy: +1 read more
  • Marcel Grünauer: I would definitely want this. I've seen goccy's talk at read more
  • moznion: Thank you for a lot of feedbacks. By the way, read more
  • Olivier Mengué (DOLMEN): +1 Very good proposal. That's exactly for this kind of read more
  • Nicholas Clark: This is an excellent proposal. The speedups demonstrated by Perl::MinimumVersion::Fast read more
  • Kirk Kimmel: I agree this is an excellent proposal. More work on read more
  • rjbs.manxome.org: This looks useful. I'd love to be able to speed read more
  • hjansen: A framework for reliable and _fast_ tokenizing/parsing of Perl code read more
  • tokuhirom [livedoor.com]: This looks good to me. read more

About this Entry

This page contains a single entry by Makoto Nozaki published on March 15, 2014 9:00 AM.

Grant Proposal: RPerl Test Suite Upgrade & Module::Compile Integration was the previous entry in this blog.

Grant Proposal: Automated generation of DWIM Perl is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.

OpenID accepted here Learn more about OpenID
Powered by Movable Type 4.38