2008Q3 Grant Proposal: Single-file GUI-enabled executable

| 12 Comments
  • Author: Vadim Konovalov
  • Title: single-file GUI-enabled executable for Windows and then for Linux w/o temporaries and dependencies
  • Synopsis: Proposal is to make a distribution of Perl program with a single-file executable much like PAR approach but without any temporaries, thus having existing filesystem untouched in any way. This executable will include Tcl/Tk into it and all Perl and Tcl/Tk core scripts will be inside this executable and will be unpacked on demand into memory.

Author
Vadim Konovalov

Title
single-file GUI-enabled executable for Windows and then for Linux w/o temporaries and dependencies

Synopsis
Proposal is to make a distribution of Perl program with a single-file executable much like PAR approach but without any temporaries, thus having existing filesystem untouched in any way. This executable will include Tcl/Tk into it and all Perl and Tcl/Tk core scripts will be inside this executable and will be unpacked on demand into memory.

Benefits to the Perl Community
Having a possibility of distributing applications is a frequently required feature. A potential user could have no Perl installed, or it could have incompatible version or lack some CPAN modules. Fortunately there are several ways to solve this problem. All these solutions are having drawbacks however. For example, PAR extracts all files into some temporary directory and then runs a program from there, thus having at least 3 un-effectiveness problems: it pollutes filesystem with temporaries (what if there's limited space on TEMP), it extracts all script files - even script files that are not actually needed (actual program run could skip), and also it creates extra subprocess, which is needed to startup properly.

There are also non-free solutions, but those are not available to everyone and so these could not be adviced as a solution to a problem.

Proposed by me approach will produce executables with following properties:

  • a single executable of about 8 Mbytes in size (this is for perl-5.10.0, earlier versions could be smaller)
  • required modules will be statically built into this executable (including Tcl/Tk)
  • this executable will have ZIP content inside it, whith all needed Perl and Tcl/Tk scripts. extraction from this ZIP will be done in-memory on the fly (when required).
  • GUI is done with Tcl::Tk CPAN module, with Tcl/Tk of version 8.5.3. Starting from 8.5.0, Tk on linux has much more attractive look-n-feel (compared to Tk8.4 on linux)

    To mention, Tcl/Tk have similar approach of creating executables (starkit, tkkit). Also all major Common-LISP implementations have possibility to produce single-file executable for easy distribution. Perl needs some efforts to be in par with both Lisp and Tcl.

Deliverables
A set of scripts to generate executables from existing programs will be created. Also, core Perl modules should be prepared for static linkage, and this part could require changes into the Perl core itself.

Project Schedule
On average I will spend every day for two hours implementing required functionality and will complete windows part within 7 weeks and linux part within another 3 weeks.

Biography
I have experience of Perl programming for more than 10 years, my CPAN ID is VKON. I also have experience in this particular area, i.e. on distributing Perl-based GUI-enabled deliverable executables based on about 5 files without temporaries.

Amount Requested
I would like to ask $2500 for windows and $1000 for linux parts of the project, so having $3500 in total.

12 Comments

Is there a reason you're not building on top of PAR? It would be a shame to have all this work done and not to reuse the infrastructure that PAR provides when it sure sounds like you're just changing the "loader" portion of something PAR already does pretty well.

I was under the impression that Windows DLLs had to be loaded from file, not from RAM. Am I mistaken?

I think Jesse brings up a good point. I remember trying to package Tcl::Tk with PAR and never quite got it working, and that is something I really wanted to do. I remember there being some technical issues with the Tcl library's configuration that made me finally give up. If it can be done, I would much rather see something like this built with PAR than without.

Yes, I am planning to reuse those PAR pieces which are applicable, or, better said, other CPAN modules which PAR uses for its task


AFAIU Module-ScanDeps was part of PAR initially, and then became independent module.

The startup ideology is much different, but even then - I could reuse Win32::Exe - but I am not sure right now

Other mentioned usefulness from my proposal is building a full-featured Perl executable in one file which will contain all core perl with some modules, much like "tclkit" at http://wiki.tcl.tk/52

Vadim hasn't discussed this plan on the PAR mailing list.

While it's entirely his decision whether he considers that worthwhile, many, many PAR users would love to be able to get by without temporaries. From my understanding, it would be possible to do away with the majority of them, but not all (shared objects using the file system to search for their siblings). Scott Stanton has made a branch of some of the PAR-related tools and implemented a mechanism that loads the .pm's from memory ("fastio" branch, for the record). He says it may not be suitable for the main branch yet, though, and doesn't have enough time to make it at the moment.

It's not like we're trying to get rid of people working on PAR. The proposal is to use pieces of PAR to write something entirely new. What's wrong with replacing pieces of PAR to make it a better tool? I think that would be much more useful to the Perl community. I'd say make a branch and work on that to become PAR 2.0 with some of the cruft removed or replaced.

It seems to me that including Tcl libraries wouldn't be *that* hard with the existing infrastructure and some hacking.

One last note: Module::ScanDeps is possibly the worst code in the PAR toolchain. It's really hack upon hack and produces lots of false-positives. That's a design decision that was made to make "pp" easy to use with little false-negatives. It ought to be rewritten using PPI. So please think twice about re-using *that* part.

That all being said, I think Vadim knows what he's talking about and is quite capable of pulling it off. I wish him good luck in the endeavour, though it may be a nightmare to support.

ScanDeps has a huge problem, it completely fails when modules are lazy-loaded on demand in an application.
This was a huge problem for the Catalyst project when they tried to include PAR support.

I like this idea. Some comments though:

Bootstrapping: Look at http://perl-compiler.googlecode.com/svn/trunk/script/pl2exe.pl for non-exe linking (think of parl.exe). This would need to be extended to scan all the modules.

Unzipping in memory makes no real sense to me performance-wise. Rather put all your code in .data segments in the exe, and simply memory map your code and modules. Much simplier and faster. The lisps you mentioned do it like this (corman, clisp, ...). Compression is achieved by final packers like upx afterwards.

Instead of the big Scandeps (static and dynamic) you can also use B::Stash at run-time.

Really only Tk? Well, it would be platform independent then, but Win32::GUI is much nicer on windows and the UI is usually preferred by standalone exe packers.

If I could vote I would definitely vote for this project.

The use case in my mind, is portable GUI scripts.

You write the script give it a GUI, pack it and send it my email to your user, no need to install anything!

Sounds good to me!

Chris Dolan: yes, DLLs could not be used from memory (although there is some TCLCORE discussion about loading of shared libraries into memory, it isn't here yet). Perl modules must be built-in statically, which makes life harder, but anyway there are benefits

rurban.myopenid.com: Win32::GUI is very possible but I am not its user. Actually Tk looks much nicer - you probably didn't follow Tk news: it allows native widgets but "tile" allows very nice looking, see http://wiki.tcl.tk/11075. We could discuss other technical details about memory map - but w/o unzip EXE would be 50 instead of 5 MBytes, isn't it so?

David Romano: I am PAR-packaging Tcl::Tk all the time, you're invited to ask questions about this, BTW Tcl::Tk has its mailing list - tcltk at perl dot org - that simple!

Steffen Mueller: I heard something about the effort of optimizing PAR so it could live w/o temporaries, but I know nothing of current status of the project.

On the other side I know my current status: I have distributable on Win32 with Perl(with core modules)+Tcl/Tk all in some 5 files: main EXE, couple of DLLs and huge 5Mbyte ZIP with
all text scripts inside it

When thinking about making it into single EXE - I kept in mind that I would study and combine and share ideas from PAR and will cooperate.

All in all, thank you all for your positive comments.

I think that the cooperation with PAR would be a great step on this project appreciation. I really do not like to reinvent the wheel. I think that is the biggest problem on Perl. People continue to rewrite, and rewrite...

Alberto
TPF GC Chair

I can imagine the Windows version being useful, but why the Linux version? I don't know of any Linux distribution that doesn't come with perl installed, the benefit of a single file executable isn't obvious to me.

Alberto Simões: I am active user of PAR, really appreciate its existence and will certainly not re-invent the wheel.

Leon Timmermans: at first, many Linux distributions do not have Tk preinstalled (also I have 2 linux devices at home that do not have perl preinstalled). at second, there are reasons why PAR works on Linux too (dependent CPAN modules, wrong Perl version, etc) - so there is a niche for this too.

Leave a comment

About this Entry

This page contains a single entry by Alberto Simões published on August 1, 2008 12:00 PM.

2008Q3 Grant Proposal: Bavl was the previous entry in this blog.

2008Q3 Grant Proposal: Improve POE::Component::IRC is the next entry in this blog.

Find recent content on the main index or look in the archives to find all content.