sounds good, but please try to build something where the logic is in the client / interfaces, not the package repository itself.
the best aspect of CPAN is that the package repository itself is just a directory tree, so setting up a custom one is trivial, and tools don't need special logic to deal with it
hi dakkar - responding here too (after we've chatted in irc) just to provide some extra details.
the base directory structure has to change a little bit to help accommodate raku's use of :ver:auth on module names but for the most part providing a mirror to the ecosystem should be very similar to cpan.
the api is not a required component for downloading or searching for modules and, as you pointed out, will be more analogous to PAUSE.
I think it's an excellent idea. The split nature of the ecosystem is a lot of trouble, and anything with an API that helps the creation of an ecosystem of tools is going to be awesome. I wholeheartedly support this.
forgot to write in my previous comment: having a repository that's just a bunch of files makes it also trivial to mirror, which helps lower upkeep costs
This proposal seems short on details; I've got several questions: what specifically do you mean by ecosystem? How will you address fault tolerance? Would the web site reuse code from metacpan or be a group up rewrite? If rewrite, what tooling? You mention that the API is limited - what specific endpoints will be available as part of this grant? What are the milestones (we just have the 2 month estimation)? What are the estimated costs for maintenance? Have you spoken with anyone at TPF outside of the grant process regarding maintenance costs or donations "exceeding operating costs" (Any discussion of donations outside this grant process needs to include at least the Treasurer)? Why "zef.pm" given the rename from Perl 6 to Raku? If this is intended to be the primary ecosystem, why does this need a plugin to zef? (Wouldn't it be a core feature?).
"ground up rewrite"
This is a very interesting proposal – that you very much.
I have a few big-picture questions about the goals for this proposal. The past few years have seen a *lot* of new language package managers, from the rise of NPM, to Golang's package manager, to Rust's Cargo. Many of these have attempted to fix perceived flaws in past package managers (though of course they could have introduced new flaws of their own). Do any existing package managers/language ecosystems serve as positive or negative examples for this project? You briefly discussed the limitations of relying on cpan for Raku packaging, but I'm curious about your thoughts about lessons we can learn from existing language ecosystems.
Another notable development in the last few years is the dramatic rise in the number of malicious packages uploaded to package repositories – including, just within the past few days, cpan (https://news.perlfoundation.org/post/malicious-code-found-in-cpan-package). I recognize that their are real limits to how much a package manager can do to protect the ecosystem from malicious packages. At the same time. most existing package managers were developed before the scale of the problem was widely recognized and I wonder if there's a way for us to lay the groundwork for better security practices while our ecosystem is still small enough for best practices to spread.
In particular, it strikes me that much of the risk comes from large dependency graphs, with many transitive dependencies, In many language ecosystems, it's easy for a developer to transitively depend on many, many packages – but it's very hard for a developer to have much visibility into what they are depending on. Could Raku do something that encourages more awareness of transitive dependencies? (Raku has a strong advantage here – we have a deep standard library and an *extremely* expressive language. Together, this means that many features that would be supplied by a package in a different ecosystem are either built-in or are trivial to hand code. Could we build on that advantage and further encourage low-dependency development?)
In any event, I imagine you have thought about all of the above more deeply than I have. I look forward to hearing your thoughts. Thanks again!
I would like to see more details about why the existing solutions either can't be fixed or aren't worth fixing as well as why this new solution is worthwhile.
On CPAN:
How will this new "dist repo" part compare to cpan in terms of: fault tolerance and simplicity. Those are the main strengths, in my mind, of the cpan option. For example: anyone can do a cpan mirror (quite efficiently with rsync), anyone can setup a private "cpan" for various reasons, and when a particular mirror is unavailable the clients have plenty of others to choose from.
I believe the only significant (read perhaps unfixable) issue with cpan is the manual pause approval process.
Another, I believe fixable, issue related to cpan is that the raku indexes in cpan are not useful and/or aren't being used. Zef is instead using an index alongside the p6c one at https://github.com/ugexe/Perl6-ecosystems. This breaks, at least, the fault tolerance aspect of cpan.
Also, exactly what about "auth and ver" does cpan not support?
On modules.raku.org:
Why can't this solution be used as a base to build on? It already has the basic functionality of a "metacpan" (docs/code view/browse, search, etc...). Perhaps it could be used to also more easily manage the p6c index.
Don't get me wrong - I'm all for more choice and improvements and such. But what is being proposed here carries along with it significant additional costs, both initial and recurring, in both funds and man hours to tpf/the raku community. And, as addressed above - at least in my opinion, isn't yet justified in sufficient detail.
I think Raku does need ecosystem improvements, and needs them badly.
This grant proposal, however, leaves me with more questions than answers.
For example, the fault tolerant ecosystem, which faults can it handle?
The website "similar to metacpan", which features are included? metacpan has (off the top of my hat, so likely incomplete) search, rendered documentation, hilighted source code view, raw source code view, stars, dependency explorer, links to issue trackers and so on.
If users are allowed to upload files, it'll need some kind of authentication, and thus either access to some sort of identity management, or implement its own. Which one will it be?
Will there be any kind of moderation features (like the options for admins to remove malicious code or spam uploads)?
What are the plans for involving the raku infrastructure operators, or really anybody else from the community, to ensure that in the end it doesn't become a platform with a single point of failure?
I ask these questions partly because depending on the answers, I might want or not want to support this project, and partly because clearly defining the scope of a project is tremendously important for a grant that is payed on completion. In fact, if you find anyway to split this grant into milestones that are useful on their own, I'd highly encourage you to structure it so that reaching one milestone can trigger a partial payout of the grant.
At Tony's request, I have amended the original post with an addendum he submitted in hopes that it will clear up some things that have been brought up (here and privately). Please check it out.