Grant Proposal: Perl::Lint - Yet Another Static Analyzer for Perl5
Sat, 15-Mar-2014 by
Makoto Nozaki
edit post
< to analyze perl source code, which is relatively slow. The processing time can be greatly shortened by about 30 times the current value by replacing PPI with `Compiler::Lexer` and `Compiler::Parser`, which are new tools created by Masaaki Goshima who authored `PerlMotion`.
Using these modules are the main ingredients to the speed up, but in my previous attempts introduce these changes to Perl::Critic, it has proven difficult to do this without changing the current codebase significantly thereby causing significant trouble for the current user base. It would especially affect the already-existing plugins for Perl::Critic.
Rather than breaking existing functionality, I propose to build a new tool that is feature-wise compatible to Perl::Critic, but with much faster processing speed.
## Benefits to the Perl Community
While Perl::Critic is an extremely useful tool, we have seen many colleagues avoiding its use because of its slowness. Improving the execution speed of Perl::Critic would lower the barrier to use in your day-to-day hacking activities, thus also leading to a better quality code, and improved development efficiency. With a fast enough Perl::Critic, one might even be able to embedded in an editor for seamless and continuous checking as you write.
## Deliverables
The new module which is compatible with Perl::Critic. And this module is much faster than Perl::Critic because it uses [Compiler::Lexer](https://metacpan.org/pod/Compiler::Lexer) and [Compiler::Parser](https://metacpan.org/pod/Compiler::Parser) instead of [PPI](https://metacpan.org/pod/PPI).
## Project Details
This project aims to create a MUCH faster version of [Perl::Critic](https://metacpan.org/pod/Perl::Critic), using [Compiler::Lexer](https://metacpan.org/pod/Compiler::Lexer) and [Compiler::Parser](https://metacpan.org/pod/Compiler::Parser)
[Compiler::Lexer](https://metacpan.org/pod/Compiler::Lexer) and [Compiler::Parser](https://metacpan.org/pod/Compiler::Parser) are tokenizer and parser for Perl5 written in C++, which offers unparalleled speed compare to pure-perl competitors used in PPI. They are used in projects such as [PerlMotion](https://github.com/goccy/perl-motion) which transpiles Perl5 code to Objective-C.
A real life example replacing PPI with Compiler::Lexer can be found in [Perl::MinimumVersion](https://metacpan.org/pod/Perl::MinimumVersion) and [Perl::MinimumVersion::Fast](https://metacpan.org/pod/Perl::MinimumVersion::Fast) . The former is implemented using PPI, and the latter uses Compiler::Lexer. A simple benchmark shows that Perl::MinimumVersion::Fast is about 33 times faster than Perl::MinimumVersion (https://metacpan.org/pod/Perl::MinimumVersion::Fast#BENCHMARK). Applying this to Perl::Critic, one can expect at least an order of magnitude faster processing.
Therefore goal of this project is to create this much faster replacement while still keeping compatibility with [Perl::Critic](https://metacpan.org/pod/Perl::Critic).
## Inch-stones
- Analyze functions of Perl::Critic (half to one day)
Extract items that Perl::Critic currently checks
- Implement tokenizer (two or three weeks)
Parse source code written in perl and divide to tokens by Compiler::Lexer and Compiler::Parser.
- Implement code analyzer (two or three weeks)
Analyze and evaluate tokens by checking with inspection rules
- Implement a system to define inspection rules (one or half week)
Implement a system to define inspection rules on this phase, as well as a basic set of rules that are frequently used (mostly ported from original Perl::Critic rules)
## Project Schedule
Period: within two months
When can I start: May 2014
## Completeness Criteria
Pass all of tests for Perl::Critic, benchmark against Perl::Critic, and release the module to CPAN.
## Bio
I am a software engineer and graduate student (majoring in high-performance computing and software engineering) in Japan. I'm experienced in systems development for about five years, and I have worked with perl for about 2 years.
I have contributed to a few open source projects (mostly involving perl) and have released a few CPAN modules. Please refer to the following web sites for details:
[https://github.com/moznion](https://github.com/moznion)
[https://metacpan.org/author/MOZNION](https://metacpan.org/author/MOZNION)
I am also a member of [Compiler::Lexer](https://metacpan.org/pod/Compiler::Lexer) development team, so I know the inner implementation of these modules which are crucial for this project. I believe I can apply those skills to this project.
## ACKNOWLEDGEMENT
While the original proposal was written by Taiki Kawakami, Daisuke Maki was responsible for editing this proposal's English.
]]>
Comments (13)
I endorse this grant.
We should really go on with Compiler::Lexer and Compiler::Parser, and a linter is the easiest way to follow.
The interpreter is a long shot, the parser is not re-entrant yet. (eval)
The proposed benefits are clear and compelling and the example of Perl::MinimumVersion::Fast gives me confidence that this project can succeed. I fully support this grant.
(If I may suggest a follow-on idea, please consider doing the work in a way that would allow a faster and more powerful Perl::Tidy as well.)
This looks very interesting. To me this is clearly the best of the submitted proposals.
(My suggestion for a follow-on would be a faster Perl::PrereqScanner, but that's another discussion).
This looks good to me.
A framework for reliable and _fast_ tokenizing/parsing of Perl code (without simultaneously executing it) would enable the creation of a whole new generation of tools.
Also the amount of money requested is (surprisingly) low for the workload.
If it turns out that either time or money won´t suffice to reach the original goals I´d suggest to focus on the low level parts: feature completeness and API/ABI of the C++ library, not Perl::Critic compatibility.
This looks useful. I'd love to be able to speed up my code that uses PPI, even if it requires porting to some new parser/DOM. This seems like a good place to explore.
I agree this is an excellent proposal. More work on these kinds of tools is healthy for the Perl ecosystem.
This is an excellent proposal. The speedups demonstrated by Perl::MinimumVersion::Fast show that applying the same approach to Perl::Critic is likely to work well. I hope that this turns out to be the case.
As everyone else is suggesting a follow-up, may I too? The CPAN testers results for Compiler::Parser suggest that currently it only builds cleanly on a few platforms. Maybe addressing portability would be a good candidate for a future grant proposal, if someone feels able to. But to be clear, please don't see those test results as a blocker for accepting or working on this grant. Compiler::Parser already passes 100% on enough platforms that things built atop it will benefit a significant part of the Perl userbase.
+1
Very good proposal. That's exactly for this kind of projects that grants should be targeted. I expect it will have wide benefits for the whole Perl 5 ecosystem.
Thank you for a lot of feedbacks.
By the way, I made a lightweight prereq scanner by using Compiler::Lexer to ensure the solution which is proposed (inspired by Mr.Leon's post).
https://metacpan.org/release/Perl-PrereqScanner-Lite
https://github.com/moznion/Perl-PrereqScanner-Lite
The new one is faster about 30 times than Perl::PrereqScanner.
Therefore Perl::Lint also can realize acceleration, I think.
I would definitely want this. I've seen goccy's talk at YAPC::Asia 2013 and it was amazing; there's little doubt that static analysis using these tools is very nice.
+1
I would love a faster Perl::Critic.
