March 2018 Archives

The Perl Conference Newsletter: 3/25/18

No Comments

In This Issue:

Keynote Speaker: VM (Vicky) Brasseur

VM (aka Vicky) spent most of her 20 years in the tech industry leading software development departments and teams, and providing technical management and leadership consulting for small and medium businesses. Now she leverages nearly 30 years of free and open source software experience and a strong business background to advise companies about free/open source, technology, community, business, and the intersections between them.

She is the author of Forge Your Future with Open Source, the first book to detail how to contribute to free and open source software projects. Think of it as the missing manual of open source contributions and community participation. The book is published by The Pragmatic Programmers and is now available in an early release beta version. It's available at https://fossforge.com.

Vicky is the proud winner of the Perl White Camel Award (2014) and the O’Reilly Open Source Award (2016). She's a moderator and author for opensource.com, a board member for the Open Source Initiative, and a frequent and popular speaker at free/open source conferences and events. She blogs about free/open source, business, and technical management at http://anonymoushash.vmbrasseur.com.

Call for Grant Proposals (March 2018 Round)

The Grants Committee is accepting grant proposals all the time. We evaluate them every two months and another evaluation period is upon us.

Maintaining Perl 5 (Tony Cook): February 2018 Grant Report

This is a monthly report by Tony Cook on his grant under Perl 5 Core Maintenance Fund. We thank the TPF sponsors to make this grant possible.

Approximately 24 tickets were reviewed, and 6 patches were
applied

[Hours]         [Activity]
  2.60          #125351 testing, sidetracked into unrelated failure, have
                reproduction problems, back to original testing, apply to
                blead
                #125351 look at minor re-work of patch, which apparently
                I’d already applied.
  5.88          #127743 review smoke results
                #127743 commit clean up, update Changes
                #127743 more Changes updates, testing, apply to blead
                #127743 reduce re-probes of the recursion limits
                #127743 more reduce re-probes (Win32 makefiles), testing
  0.37          #130841 (sec) review
  2.82          #131844 (sec) research
                #131844 (sec) research, comment
                #131844 (sec) request CVE ID
                #131844 (sec) testing patches against branches
  0.37          #132062 (sec) review overnight discussion
  9.14          #132063 (sec) catch up on discussion
                #132063 (sec) testing, research and comment
                #132063 (sec) comment
                #132063 (sec) more testing against branches, #p5p
                discussion of an AIX issue
                #132063 (sec) try to apply patches, testing
                #132063 (sec) more try to apply patch against maint-5.24
                #132063 (sec) comment
  6.62          #132227 (sec) research, comment
                #132227 (sec) testing, comment
                #132227 (sec) rebase blead, apply new patch, testing
                #132227 (sec) debugging
                #132227 (sec) more debugging
                #132227 (sec) comment
  0.98          #132683 debugging, produce a fix and comment with patch
  3.19          #132833 debugging, work on a fix, testing
                #132833 review, comment briefly
                #132833 review code, review fix and close
  0.25          #132834 research
  0.40          #132843 review discussion and related ticket and comment
  2.47          #132849 research, fix, testing on linux and freebsd, apply
                to blead and comment, fix warnings.pm porting test failure
                #132849 review comments, testing on Win32
  0.22          #132866 testing, comment
  1.70          #132867 testing, comment, find an old related ticket, test
                old patch, comment with patch
                #132867 testing, apply to blead
  0.77          #132870 testing, try to reproduce outside of perl, work on
                a fix and comment
  2.10          #132876 testing, work on a fix
                #132876 more testing, comment with patches
  8.59          #132893 testing
                #132893 testing, debugging, comment
                #132893 debugging
                #132893 debugging, work on a workaround
  0.58          #132920 debugging, comment
  0.98          more security patch wrangling
  0.25          push security patches to security repo, let pumpkin know
  0.63          reply Dave’s Storable email
  1.82          review security queue to select security issues
  3.60          security patch round up
  0.63          security patch wrangling
======
 56.96 hours total

As has been announced previously, TPF is working on a process to allow for more public participation in adding new board members. I'm happy to present Ricardo Signes as our first new board candidate nominated with the new process. Ricardo (aka RJBS) is well known to the Perl community as a former pumpking and he offers some addition background in his bio below.

Ricardo Signes spent four years studying philosophy and religion before learning that jobs in those fields did not abound. He managed to convert his hobby into his profession, convincing his employer that it was a good idea to pay him to create software in Perl. Since then, he has become a prolific CPAN author, part of the "Toolchain Gang" that maintains the PAUSE and CPAN infrastructure, and spent five years as "Pumpking," the Perl 5 project manager. Ricardo is currently CTO at FastMail, where he manages a team writing and maintaining free and proprietary software, much of it written in Perl 5. In his free time, Rik writes code to help making writing code easier.

And his answers to a few questions on joining the TPF board:

  • Why do you want to become a member of the TPF Board?
  • What existing TPF initiative(s) are most important to you and why?

I'd like to help TPF accomplish its mission of supporting the Perl programming languages, and especially to help increase understanding of what TPF does. I think I have a good knowledge of the Perl community, and can help provide guidance on the board, but most especially I want to help make it easy to find out what TPF is doing.

The TPF's most visible work is its grants work and conference support, and those are probably the ones I think are most valuable, too.

  • What goals would you like to see TPF pursue in the future?

Outreach programs are hard and time-consuming, because they require a good combination of project, mentor, and newcomer to succeed, but I'd like to look into backing projects like those, and then share how those efforts succeed or fail.

For the next two weeks, we welcome constructive community discussion and feedback before the board continues with the process.

The Perl Conference Newsletter: 3/11/18

No Comments

In This Issue:

Round 2 Talk Submission Notifications Sent

No Comments

Round 2 CFP notifications have been sent! Check your email if you submitted by 2/26/2018. Notifications sent from [email protected]

Round 3 CFP is still open! https://goo.gl/forms/I5a5XneNsm4Vq93D2

TPCiSLC

Timo has made good progress and is writing about it. Please see: Delays and Delights


Delays and Delights

Hi, my name is timotimo and I'm a Perl 6 developer. I've set up this blog to write reports on my TPF Grant.

Before the actual report starts, I'd like to issue an apology. In between my grant application and the grant being accepted I developed a bit of RSI that lasted for multiple months. I already anticipated that near the end of January a move would occupy me a bit, but I had no idea how stressful it would turn out to be, and how long afterwards it would keep me occupied.


Photo by Lyndsey B / Unsplash

I regret that it took me so long to actually get started. However, I've already got little bits to show for the first report of this grant!

Delight №1: Non-crashy multi-threaded profiles

Until now if you've added a signal handler for Ctrl-C to your program, or used run, shell, or even Proc::Async or any async IO, rakudo or moarvm will have started an additional thread for you. Even if you don't necessarily realize it, this caused the profiler to either violently explode, or return useless or partial results.

Just these past few days I've made a bit of reliability work for the profiler available to everyone running rakudo from the latest source code. Now the profiler will only sometimes abort with the error message "profiler lost sequence" – a bug that I'm also going to chase down as part of the grant. On top of that, the HTML profiler frontend now shows timing and allocations from every thread.

Delight â„–2: Faster Heap Snapshots

N.B.: I have done the work discussed in this section before the grant actually started, but I think it's still interesting for you.

As you can tell from the – sadly mildly messed-up – grant application, I also work on a second profiler: The Heap Snapshot Profiler. It takes a snapshot of the structure of all objects in memory, and their connections. With this information, you can figure out what kinds of objects account for how much of your memory usage, and for what reason any given object is not being removed by the garbage collector.

If you've already tried out this profiler in the past, you may have noticed that it incurs a significant memory usage increase during run-time. After that, it takes a surprising amount of time to load in the heap snapshot analyzer. This changed noticeably when I implemented a new format for heap snapshots.

Until then, the format had one line (i.e. \n delimited) per snapshot (i.e. for each GC run) and a json-encoded list of strings up front. The snapshots themselves were then a few lines with lists of integers.
This caused two little issues in terms of performance:

  • Encoding the list of strings as JSON had to be done in NQP code, because of the somewhat complex escaping rules.[1]
  • Reading the heap snapshot requires a whole lot of string splitting and integer parsing, which is comparatively slow.

The binary-based format I replaced it with addresses both of these issues, and has a few extra features for speed purposes:

  • The list of strings is written as a 64 bit integer for the string length followed by the string itself encoded in utf-8. At the very front is a single 64 bit integer that holds the list's size.
  • Snapshot data is encoded with either 64 bit integers or a very simple variable-sized integer scheme. Each list stores its size at the beginning, too.
  • The very end of the file contains an index with the sizes of all the sections, so that skipping to any point of interest is cheap.

Armed with the index at the end of the file, the binary format can be read by multiple threads at the same time, and that's exactly what the new Heap Snapshot Analyzer will do. In addition to the start positions of each section, there's also a pointer right into the middle of the one section that holds variable-sized entries. That way both halves can be read at the same time and cheaply combined to form the full result.

The "summary all" command loads every snapshot, measures the total size of recorded objects, how many objects there are, and how many connections exist. The result is displayed as a table. Running this against a 1.1 gigabyte big example snapshot with 44 snapshots takes about a minute, uses up 3â…“ CPU cores out of the 4 my machine has, and maxes out at 3 gigs of ram used.[2] That's about 19 megabytes per second. It's not blazing fast, but it's decent.

Delight â„–3: Web Frontend

Sadly, there isn't anything to show for this yet, but I've started a Cro application that lets the user load up a heap snapshot file and load individual snapshots in it.

It doesn't seem like much at all, but the foundation on which the other bits will rest is laid now.

I intend for the next posts to have a bunch of tiny animated gif screencasts for you to enjoy!

Delight â„–4: Faster Profile File Writing

The SQL output will likely be the primary data storage format after my grant is done. Therefore I've taken first steps to optimize outputting the data. Right now it already writes out data about 30% faster during the actual writing stage. The patch needs a bit of polish, but then everyone can enjoy it!

Next steps

The next thing I'd like to work towards is trying out the profiler with a diverse set of workloads: From hyper/race to a Cro application. That will give me a good idea of any pain points you'd encounter. For example, once the ThreadPoolScheduler's supervisor thread runs, it'll spend a majority of its time sleeping in between ticks. This shows up very prominently in the Routines list, to name one example.

Of course, I take inspiration from my fellow Perl 6 developers and users on the IRC channel and the mailing lists, who offer a steady supply of challenges :)

If you have questions, suggestions, or comments, please feel free to ping me on IRC, or find this blog post on reddit.

Thanks for reading; see you in the next one!
- Timo


  1. It could have been done in C code, but it feels weird to have JSON string escaping code built into MoarVM. ↩︎

  2. Figuring out why the ram usage is so high is also on my list. Amusingly, the heap analyzer can help me improve the heap analyzer! ↩︎

This is a monthly report by Dave Mitchell on his grant under Perl 5 Core Maintenance Fund. We thank the TPF sponsors to make this grant possible.

I have spent the last month mainly doing two things:

Fixing various issues with overloading and the new multiconcat op. I
decided that rather than continue adding more and more special-case
handling to the existing code, I would handle the overloading in
pp_multiconcat() in a different way which avoids most edge cases.
This work is complete.

Making the parser die more gracefully if an attribute is encountered
*following* a subroutine signature (they were recently moved to precede
it).  Currently the parser tends to just spew a whole bunch of confusing
warnings and errors.
This work is also complete.

SUMMARY:
      9:11 RT #132760 BBC YANICK/List-Lazy-0.3.0
      4:11 RT #132772 Blead Breaks CPAN: TIMB/Geo-StreetAddress-US
     25:34 RT #132783 Blead Breaks CPAN: ADAMK/SQL-String-0.02.tar.gz
      3:57 RT #132875 Blead Breaks CPAN: ZDM/Pcore-v0.56.5
      1:00 ext/File-Glob/t/rt131211.t: fix timing issues
      0:41 fix noisy tests
      2:55 process p5p mailbox
      0:24 tr///c Deparse test failure on EBCDIC
    ------
     47:53 TOTAL (HH::MM)

 228.4 weeks
3010.3 total hours
  13.2 average hours per week

There are 123 hours left on the grant.

Grant Report : Complete YAML::PP - February 2018

No Comments

Tinita spent about 20 hours coding YAML::PP this month.

To her growing YAML bibliography, she has added "Strings in YAML - To Quote or not to Quote". Here Tina shows very clearly that Ingy's ultimate goal was to write the Kobiyashi Maru of format specifications. (She doesn't say that.)

In "Safely load untrusted YAML in Perl", she compares multiple parsers with respect to object loading and cyclic reference handling.

She also continued to hack away on YAML::PP. You can read about that in her latest report.

MAJ

January 2018 Grant Votes

No Comments

The Grants Committee has concluded the voting of the January 2018 round.

There were six proposals in this round.

This is the report from zefram on his Perl 5 grant under the Perl 5 Core Maintenance Fund for February 2018. We thank the TPF sponsors to make this grant possible.

The hours that I have worked in 2018-02 pursuant to my TPF core
maintenance grant are as follows.

 10h47m  [perl #132894] Blead Breaks CPAN:
         MAROS/DateTime-Format-CLDR-1.19.tar.gz
  8h52m  review mail
  3h08m  [perl #132833] COW bug in :encoding layer
  2h45m  [perl #132817] Blead Breaks CPAN:
         SUNNAVY/Net-Google-Code-0.19.tar.gz
  2h32m  [perl #132900] Blead Breaks CPAN:
         FELIPE/Crypt-Perl-0.17.tar.gz
  1h31m  [perl #132385] BBC: Commit e839e6ed breaks Bit-Vector-7.4
  1h23m  [perl #132822] Blead Breaks CPAN:
         JETTERO/Games-RolePlay-MapGen-1.5008.tar.gz
  1h10m  [perl #132910] Blead Breaks CPAN: Test::MockObject
    45m  PathTools version portability
    42m  [perl #132875] Blead Breaks CPAN: ZDM/Pcore-v0.56.5.tar.gz
    39m  [perl #132902] Blead Breaks CPAN: Class::Std
    36m  [perl #132788] Blead Breaks CPAN:
         LEMBARK/Object-Trampoline-1.42.tar.gz
    33m  [perl #132760] Blead Breaks CPAN:
         YANICK/List-Lazy-0.3.0.tar.gz
    30m  [perl #132834] Blead Breaks CPAN:
         SMUELLER/ExtUtils-InferConfig-1.04.tar.gz
    29m  [perl #132873] Blead Breaks CPAN: JOHNH/Fsdb-2.64.tar.gz
    28m  [perl #131046] [PATCH] Carp: Do not crash when reading
         @DB::args
    20m  [perl #7515] perlio bug ?
    20m  [perl #132876] ppport.h breaks croak_xs_usage() on old Perls
    17m  [perl #132828] Tricky code can bypass Carp overload
         protections and trigger exceptions
 37h47m  TOTAL

At the end of the month there was 32h20m of authorised working time
remaining in this grant.

About TPF

The Perl Foundation - supporting the Perl community since 2000. Find out more at www.perlfoundation.org.

About this Archive

This page is an archive of entries from March 2018 listed from newest to oldest.

February 2018 is the previous archive.

April 2018 is the next archive.

Find recent content on the main index or look in the archives to find all content.

Pages

OpenID accepted here Learn more about OpenID
Powered by Movable Type 6.2.2