mirror of
https://github.com/semver/semver.git
synced 2026-07-10 19:50:47 -05:00
[GH-ISSUE #567] Looking for representative SemVer string databases. #5529
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: github-starred/semver#5529
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @jwdonahue on GitHub (May 18, 2020).
Original GitHub issue: https://github.com/semver/semver/issues/567
@isaacs, @anangaur, you represent the two largest caches of SemVer version strings, can you assist me with this?
I am trying to look at the distribution of character counts in each of the five fields and the distribution of prerelease/meta identifier counts. I'll find a place to publish all the results and point to them from this thread eventually. I think there's a lot of code out there for processing SemVer strings that is non-optimal, and knowing the global distribution of character counts could help tool makers make more informed decisions.
I've already mined one small database used to track rust buckets (err..um..crates?) and found the following distributions (based on this 3 week old data) from 11152 version strings:
If anyone has a database of SemVer version strings they can dump to a file and send my way, I'd really appreciate it. For the smaller online data sets, just point me to the correct UI/API's and I can probably run the queries myself, but the larger ones are going to be problematic for me, as I am currently running tethered to my phone here on the Navojo Nation, and will be for months. I can post process any compressed text data into a CSV for analysis. I don't need package names or any PII.
If it's gigs of version strings, then I'll need counts of unique strings or just the field distributions to reduce the data load. I'll post a C# program on my github account shortly.
If there's any volunteers who have fat enough pipes to gather/summarize this data for me, that would be most appreciated.
@ljharb commented on GitHub (May 18, 2020):
I'd suggest including absolute numbers as well;
0.08%of npm, for example, is still a massive number.@jwdonahue commented on GitHub (May 18, 2020):
Project repo: https://github.com/jwdonahue/SemVerStatsAnalysis
References:
theory/pg-semver#52
jwdonahue/SemVerStatsAnalysis#1
@Seldaek commented on GitHub (May 19, 2020):
Here you go, the 1.7million versions (I excluded dev ones as they represent branches and are thus not very relevant here) from https://packagist.org
versions.zip
@jwdonahue commented on GitHub (May 19, 2020):
Sweet! Thank you. I would actually like to see the dev strings as well. Part of what I am planning on doing is documenting the most common types of errors in usage.
@Seldaek commented on GitHub (May 19, 2020):
The dev ones are a bit of a particularity in composer/packagist as they are automatically created versions tracking branches, e.g. a 1.10 branch creates a
1.10.x-devversion, while the master branch or any non-numeric branch results in dev-branchname so dev-master etc. These are most definitely not semver and also not really part of the releases of the projects, so as such I don't really think they're relevant for you? All tagged releases are included in the file. We do not allow using thedevsuffix in tags as this would conflict with our automatic versioning of branches. You'll find RC/beta/alpha/patch stuff in there though.@jwdonahue commented on GitHub (May 19, 2020):
Okay, I am already aware of the -dev pot-hole, but the other stats
actually matter to me. Would it be possible to randomize the PII
portions of the strings? I can make adjustments to my tooling
https://github.com/jwdonahue/SemVerStatsAnalysis and make that
available to you.
If they were using -a.dev, or some other sensible tag, then these would
be legitimate strings that a high-rate shop might produce and therefore
the stats on sizes of strings are important for tool
developers/maintainers. I've worked in many shops where dev bits seem
to always find there way off the dev's box, so while not intended for
public consumption, they are often effectively published, or all they'd
really need is a devName-timestamp for a version.
Correction: devName-ticket#-timestamp version number.
@Seldaek commented on GitHub (May 20, 2020):
I'm really not quite sure what you are saying/asking, but if it helps here is the full unfiltered data set of 2.2million versions
versions.zip
@jwdonahue commented on GitHub (May 24, 2020):
Okay, here's some results from the data so far:
Results.zip
@jwdonahue commented on GitHub (May 24, 2020):
Rust crate distributions:
@jwdonahue commented on GitHub (May 24, 2020):
@jwdonahue commented on GitHub (May 24, 2020):
So basically, a majority of the version triple fields are single digit, with patch having a broader range out to 20 digits, but still weighted heavily in the 1 & 2 digit range. With only the two sources of data at this point, I am seeing some interesting variations in the non-semver strings. A very large set of 'v' prefixed strings that would otherwise qualify as SemVer and lots of single and double field numeric versions as well as some mostly alpha strings.
I'll try to characterize the non-semver strings next.
@ljharb commented on GitHub (May 24, 2020):
A semver version with a preceding v is still valid; are you not counting those in these numbers?
@Seldaek commented on GitHub (May 24, 2020):
Yeah at least in the packagist dataset, the
vprefix is completely ignored when normalizing versions before doing dependency resolution, so it probably should be ignored too before counting anything.@jwdonahue commented on GitHub (May 25, 2020):
The 'v' prefix was part of SemVer 1.0.0 and -beta, but only for use in version control systems (ie; git tags). No idea why they thought that it was ever a good idea, but it was removed in SemVer 2.0.0.
So, no I do not count v1.0.0 as a legal semver string. I did mark it as a near-miss and then automatically processed the cleaned up near-misses in a separate step.
I suffered some angst over the decision to relegate the SemVer 1.0.0 strings to the non-semver category. The 'v' designation was never unique to SemVer and only allowed in a special class of tooling. Therefore, it is my view, that it was an implementation detail that should never have been allowed in the spec to begin with. But that's not as relevant as the fact that I am not processing VCS tags, I am working with SemVer strings, and those clearly do not include the 'v' prefix, as-per all of the published SemVer specs.
One thing that SemVer has lacked from the beginning is a means to disambiguate SemVer v1 from v2 strings or from similar looking strings that are not based on the specified semantics. I have taken the position that v1 is essentially defunct/useless, by virtue of the existence of v2. The spec doesn't say that explicitly, but the lack of any robust means to disambiguate between them, suggests no other interpretation.
@ljharb commented on GitHub (May 25, 2020):
@jwdonahue sure but if your goal is to determine usage, you should be liberal in what you accept, because it's not particularly useful to gauge usage of strictly spec-compatible versions.
@jwdonahue commented on GitHub (May 25, 2020):
I did include the near-misses and intend to expand on not just identifying those, but also stats on the entire range of version strings that I can find in the wild. I just treat them as a special class. The results are all keyed to the data source, so combining them for a look at the 'big picture' shouldn't be too difficult.
It's a prototype tool developed during my 7 mo. old grand-daughter's increasingly infrequent naps. My daughter is a pediatrician on the Navajo reservation, and I am her only viable child-care option at the moment.
I'll improve on it later. What I need right now is data from more sources, so I can get a better idea of the range of "almost semver" and all the other non-semver variations. I'll also add prerelease and meta token counters (# of alpha, beta, rc, etc) and create a dictionary of most common symbols.
By end of next month, I should have a fairly descent internet connection at the house. Currently tethered to my cell phone in a very low service availability/quality area.