mirror of
https://github.com/semver/semver.git
synced 2026-07-11 04:52:47 -05:00
[GH-ISSUE #483] Discussion: pre-release versioning and PEP440 cross-compatibility #7331
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: github-starred/semver#7331
Reference in New Issue
Block a user
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @IvanAnishchuk on GitHub (Jan 5, 2019).
Original GitHub issue: https://github.com/semver/semver/issues/483
Some may be aware that Python has its own specification for version format, PEP440. Of course, it has goals and context different from that of SemVer and it notably doesn't specify the meaning of any version components or incrementing rules, but here I'm going to try to compare them in parts relevant for version string parsing and determining the precedence. My hope is to identify the format subset that can be compatible with both specification, and possibly propose some slight improvements for one or both of them that would help with cross-compatibility. I also make a few references to other standards and common practices to illustrate common patterns (but I don't claim that I know everything there is to versioning, of course).
Unlike SemVer, PEP440 doesn't have too many restrictions on the public version component (basically any sequence of dot-separated numbers would do) which makes that part compatible with SemVer and PEP440 actually recommends using the SemVer spec. But, on the other hand, it doesn't allow as much freedom for the pre-release component and specify more complex precedence rules that are not always lexicographical (and it also has a completely incompatible post/rev component, more about that later). Then, SemVer has plus-separated build metadata and PEP440 has plus-separated local version label which are largely the same thing (a bunch dot-separated numbers or identifiers) but have different precedence rules.
1. First of all, to comply with SemVer format the public version component must consist of exactly three numeric components and neither of the three can contain leading zeros. Precedence rules are obvious and compatible with most tools. Ok, now we have compatible public version format, that's a very good start.
(Actually, why not leading zeros? I mean I don't like them much but I can see a use for zero-padded numbers so that versions can be compared without any parsing, as if they were arbitrary strings... Yes, it would definitely be an ugly hack but in some hostile environments that could come in handy. There might've been some reasons for that part of SemVer spec but I don't understand it.)
2. Pre-release component is defined very differently in the two specifications in question. It's the opposite situation from what we have for public version part. SemVer only requires that it's separated from public version component with a hyphen and consists of dot-separated series of alphanumerics and hyphens. The PEP, on the other hand, specifies four possible suffixes (most of which can have some alternative spellings too) that can be followed by a number. These suffixes are (in the order of precedence): dev, a (aka alpha), b (aka beta), rc (aka c). They cannot be used together except for the dev one (which has to be the last component there if present).
Precedence rules are very different for this part. First step to compatibility at least for alpha-beta-rc releases would be to only use the following format: 1.0.0-a.1 (alpha), 1.0.0-b.1 (beta), 1.0.0-c.1 (release candidate). It's not the canonical form for PEP440 but it is compatible and it will ensure correct precedence according to the SemVer rules. Using
cvsrcspelling is a judgement call (cform is not recommended and arguably is less common in public projects but it's the least surprising if you have to resort to lexicographic comparison, a < b < c). Using numbers is mandatory (unless there's only one alpha, beta, or rc ever, which is uncommon) because SemVer doesn't prescribe any normalization rules as is using the the dot between suffix and the number (otherwise numbers will be sorted lexically which is not a good thing if they could possibly surpass 9). These a/b/c suffixes are enough for many projects' pre-releases so having them sorted is a very good start.3.
devreleases break pretty much all simple precedence rules. They are meant for things like nightly builds or early builds made directly from project's source control and so on (unlike a/b/c that are meant to be stages of preparing a final release or some such). The dev suffix can be added to about any release (final, a/b/c pre-release, or post-release) and must be sorted before anything else (1.0.0-dev.1 < 1.0.0-a.1.dev.1 < 1.0.0-a.1 < 1.0.0). Simple lexicographic comparison will break if they are used together with any other pre-releases, dev releases of a pre-release should also be sorted before the pre-release itself which breaks the length rule ("A larger set of pre-release fields has a higher precedence than a smaller set, if all of the preceding identifiers are equal"). The only way to solve this consistently that I can see is to add it as a special case in the next SemVer spec if the community agrees that this is worth it, otherwise dev releases can't be used with any other types of pre-releases with consistent ordering. Perhaps dev releases could use a different separator so they are visually different from a/b/c pre-releases and won't be confused by neither humans nor parsers? (PEP440 allows using dot, underscore, and no separator... underscore could possibly work, it's also the separator used in pre-releases for perl modules, for example, but this needs further discussion)4. Even more complicated case is post-releases which SemVer knows nothing about. They also look very similar to pre-releases, enough to be confused with each other, but their precedence rules are very different: post-releases must be sorted after the corresponding final releases (or corresponding pre-releases, a/b/c pre-releases can have post-releases too, although PEP440 discourages actually using them recommending to make separate pre-releases instead when needed). To extend the previous example: 1.0.0-dev.1 < 1.0.0-a.1.dev.1 < 1.0.0-a.1 < 1.0.0-a.1.post.1.dev.1 < 1.0.0-a.1.post.1 < 1.0.0 < 1.0.0-post.1.dev.1 < 1.0.0-post.1 And just like dev releases consistent sorting for SemVer here cannot be achieved without updating the spec, but even more so: post releases should be sorted after the corresponding final release, not before, while SemVer 2.0 doesn't distinguish them at all and would instead consider them just another pre-release. Is their meaning or function important enough for them to be added to the SemVer spec? The spec itself recommends only using them for "addressing minor errors in a final release that do not affect the distributes software", however my guess is that more often projects use them for maintenance and bugfix releases (practice mentioned and discouraged in the spec). I'm not sure whether that is a valid use case (it certainly is in some of the projects I work on), the only usable alternative is to increment the lowest public version component instead which sometimes can be complicated if you are limited to three components only (i.e. the number could already be taken, and simply using x.y.z+1 where x.y.z is the last x.y.* release might not always be appropriate, depending on the situation) so I'd say it is a good thing to have for occasional use. PEP440 allows for several alternative spellings for post-releases, it can be indicated by a suffix such as
post,rev, orrfollowed by a number (optional, defaulting to implicit 0) and can optionally use dot, hyphen, or underscore as separator. When separated by a hyphen, the suffix can be optional (but in this case the number is not), it looks like 1.0.0-1 (it is also the format some package management systems use -- they have to follow upstream versions and certainly can't increment the micro component themselves, one of the common situations when post-releases/revisions are required -- I'd say it is a very popular way to specify revisions, another good one that I know of is 1.0.0-r1 used in Gentoo, some systems also add their own suffixes like netbsd would use 1.0.0nb1... but I guess those naturally belong in build metadata, except see (6) about sorting builds).Again, if somebody besides me thinks that dev and post releases should be added to the SemVer spec, perhaps using underscore is a possible way to avoid clashing with any existing pre-releases (as both dev and post releases have special precedence rules it might make sense for them to look differently from pre-releases). Underscore is currently not allowed to be used in SemVer versions, so it could be introduced in a fully compatible manner. Or, perhaps, underscore should be only used for one of them and some other separator for the other, this needs discussion and careful consideration. A few examples of what such versions might look like using underscore as separator: 1.0.0_dev.1, 1.0.0-a.1_dev.1, 1.0.0-a.1_dev.1, 1.0.0_r.1_dev.1, 1.0.0_post.1, 1.0.0_rev.1, 1.0.0_rev.1-dev.1, 1.0.0_post.1, 1.0.0-a.1_post.1.dev.1 -- dev and post components are always the last components (between pre-release components and build metadata if present) so we might possibly define the underscore separator for post-releases like hyphen for pre-releases and allow having an additional pre-release-like section after that so we can use dev (except that dev can be used on its own as a normal-like pre-release but in that case it should be sorted before other pre-releases... okay, my head starts spinning, I don't think I can make a decision on this right now, a further discussion is required).
As another alternative, it also might feel easier to adjust sorting rules for identifiers in what is currently called pre-release component (we might call it something else if it contains post-release info as well) so that they start having semantic meaning instead of being arbitrary series of letters and numbers anymore (I've seen some interest to this topic in other issues). It might break compatibility, yes, but I doubt that anybody ever used pre-releases like 1.0.0-rev.1 or 1.0.0-post.1 and actually needed them to be sorted before the 1.0.0 or that anybody ever used, for example, both
alphaanddevkeywords in their pre-releases (for the same final release) and would be affected in a bad way by the change in their precedence (most likely that in both cases, if it ever happened with anybody at all, they meant for them to be sorted in a way that is not currently prescribed in SemVer and simply used non-compliant custom tools or implemented some other workarounds, or perhaps it didn't matter to them, one of the projects I work on tries to use SemVer and does, in fact, have post-releases, nothing is sorted correctly there and we simply don't care about it, it would however prevent us from using SemVer-compliant tools for automation if we ever wanted to, the current piece of software that merges branches and assigns version number to releases is customized -- the versioning scheme there is not compliant with PEP440 either, but that's another story). Anyway, adding semantic meaning for components in the pre-release section might actually be more in the spirit of SemVer if not quite the letter of the current version. I'm not sure whether listing all the possible keywords/formats with all their special meaning and complex sorting rules in the SemVer spec is appropriate (after all, being short is one of the advantages it has), but some of those keywords are established common practices in the open-source software community so it is possible that everybody would benefit if they are somewhat better formalized. We don't have to limit ourselves to the particular set of those keywords I listed here, as long as we avoid deliberate contradictions we can choose a larger or a smaller set if we decide to. We can probably skip the restrictions on combining certain keywords (pythonic versions can't have both alpha and beta components, for example, we can totally have it allowed in SemVer, and just like with other things I mentioned above anybody who cares can always decide to choose a compatible subset of available options), equivalency of spelling variations like a/alpha is desirable but is optional as well (most projects enforce a single spelling anyway), incompatibility of pre-releases that only have a single numeric component could stay (even though it is somewhat confusing and I doubt its usefulness).Essentially, all that needs to be changed is stating that certain keywords in the hyphen-separated suffix part should be sorted before anything else (dev), others must follow the previous lexical/numeric sorting rule (with all of them still sorted before nothing), and certain keywords should be sorted after nothing (post, rev, r, possibly something else). This option has the advantage of keeping the format visually the same. Basically the decision has to be made between changing the looks or changing the sorting rules incompatibly for a few special cases.
For both dev and post releases, if they are ever introduced in SemVer (or perhaps a hypothetical forked spec which I'm not working on and most likely never will) it's not that important to port all the details from the PEP, the recommended SemVer form for these things doesn't have to look like canonical format in the PEP, it might be more or less flexible it that makes more sense, as long as there is at least one way to specify them that is compatible with both specs, that tools for both formats can parse, and precedence works the same way in both worlds. Because these are two documents from different worlds: one is trying to describe established practices in a single community using a single programming language and the other is proposing the way everyone should version their stuff. But it would be great to build a bridge that could connect them when needed without breaking anything... Anyway, one more thing.
5. Version epoch is the third and last thing from PEP440 that SemVer can't possibly support in it's current version but maybe it should. It's extremely useful for switching between different versioning schemes e.g. from date-based versions (or some other non-semantic versioning with dot-separated numbers) to SemVer and back. In the PEP format they should look like this: 1!1.2.3 -- a number preceding the rest of the version, separated by an exclamation mark. If omitted (and of course most projects do omit it... at least at first :) ) it defaults to zero, so an explicit epoch number is only required when you change the versioning scheme or, perhaps, reset your major version, or whatever. I'm not aware of any other mechanism that would allow you to switch to SemVer from, say, date-based versions without it all looking weird or removing your public releases or renaming your whole project or something equally impractical. And at least this part can be implemented in 100% backward-compatible way and, in fact, might make adoption of SemVer easier for some existing projects.
6. Build metadata is defined as unsortable in the current SemVer spec. This is a relatively minor thing, but I don't understand the reasoning behind this rule. True, some kinds of build ids can't be sorted consistently (like hashes or any random data) but some totally can (like sequential build numbers or timestamps). I don't see anything breaking if this were changed from undetermined order for versions that are the same except for the build part to a defined order using the same algo as for pre-releases (each of dot-separated components sorted lexically if they contain letters, numerically if they are numbers, numbers < non-numbers, longer sequence < shorter if the preceding components are equal). I guess some tools might prefer to keep comparing using the current rules (ignoring build metadata), and some probably already have optional comparison that sorts builds as well. Perhaps there should be more than one comparison algo defined, one including the build and one ignoring it because sometimes finding the most recent build of some release is quite important (or, at least, using the most recent build is not worse than choosing one at random) while sometimes the build indeed doesn't matter (in which case it doesn't matter how they are ordered as long as there is a way to establish that two versions are equal except for the build id). I'd suggest including build metadata for inequality comparison (as in "which version/build is the most recent?") and ignore it for equality comparison (as in "is this version equal to that one?"). Actually, having some defined order for the builds might be even easier to implement than not having one, sometimes at least.
I also must add in conclusion that post-releases/revisions and epochs are not concepts specific to the pythonic world, package versions in both Debian and Red Hat, for example, have both of those even if the particular format for them varies somewhat from system to system. It would be understandable if somebody decides to introduce them in SemVer but with format incompatible with PEP440 (but, perhaps, more popular in some circles or better-looking or whatnot) and although I certainly wouldn't like that just out of my own selfish interest, it could be an argument for revising PEP440 and bringing compatibility from the other side, spelling differences should hopefully be easier to resolve than conceptual ones...
Admittedly, I haven't done too much research on alternative standardized version formats, if there's anything that's worth reading on the topic I would love to see some links. Yes, I'm aware of this one, however it breaks SemVer spec in ways I don't particularly like. Some parts of it could still be useful as could be this
@alexshpilkin commented on GitHub (Feb 7, 2019):
@IvanAnishchuk writes:
The reason, I guess, is to avoid ambiguity with some (mostly old) version schemes that used a two-component
MAJOR.MINORtreated as a decimal fraction. That meant, for example,3.0 < 3.01 < 3.02 < 3.1 = 3.10.@jwdonahue commented on GitHub (Mar 31, 2019):
@IvanAnishchuk, I am still absorbing your full text, but I want to address:
The semver spec specifies a fuzzy partition between numerical fields and strings. Obviously it's all about version strings, but it then constrains how they should be expressed and parsed. I believe the original spec allowed leading zeros without saying so, but then they were expressly forbidden. It may be the later had something to do with the packaging tool owners, some of whom, apparently thought it was too difficult to convert "001" to 1, or that there might be some ambiguity. It may in-fact be that there are some cultural issues involved. Generally, the spec has erred on the side of implementation simplicity and compactness. There is a moral risk to allowing leading zeros, in that some tools might actually require them, because they are easier to sort as strings than having to first convert them to integers.
I suspect the real reason however, has to do with the fact that if you allow leading zeros, and pure string sorts, you will have folks who use them and then do not provide enough width in each field to accommodate a full product life cycle. What should they do when they run out of digits? Who decides how wide each field should be? It's probably easier to say "no leading zeroes because we do not want you tool makers to ever think it's okay to sort these fields as strings and lock your users into excessive leading zeroes to provide life-cycle requirements".
The impact of this decision on users of SemVer is trivial and I would argue almost as trivial for implementers.
@jwdonahue commented on GitHub (Mar 31, 2019):
Here's all the history on this that I could find here at semver/semver.
@jwdonahue commented on GitHub (Mar 31, 2019):
As for the rest of your post, have a look at VersionMeta and VersionSchema, I don't think any of the SemVer stake-holders, including me, are interested in SemVer blending-in with PEP440, especially given that PEP440 recommends SemVer in its current form, over all the alternatives.
@IvanAnishchuk commented on GitHub (Apr 16, 2019):
It doesn't, in fact. It encourages to abide by it for the "Major.Minor.Patch aspects" which probably is a fine distinction but an important one. It doesn't recommend the incompatible parts, I mean.
Not sure what exactly you meant by this (it's probably not something anybody wants) but examining conceptual differences and design choices is useful nevertheless. If some conceptions are so bad that they must never ever find their place anywhere near the semver spec we might wanna be aware of them and maybe consider avoiding them elsewhere as well. Some, on the other hand, are already used with semver-versioned packages and it might be worth it to standardize a practice or two (guess what happens in e.g. debian package repository when some package foo switches to semver and suddenly goes from
foo-123.123lambdatofoo-1.0.0? I think switching from and, more importantly, to semver is important enough to be worth some discussion. As is the question of what to do when you run out of major numbers -- apart from feeling irresponsible and short-sighted, that is :) ).@jwdonahue commented on GitHub (Apr 20, 2019):
There's an infinite number of positive integers, how could you ever run out of them?
@RookieRick commented on GitHub (Feb 14, 2020):
On a practical level, I'll add that as a relative (~1-year) Python newbie, I just spent an hour or so of valuable gaming time 😂researching why my CI was rejecting my patch because I tagged it
4.0.0-rc.0, learning thatsetuptoolswas "normalizing" it to4.0.0rc0, messing with monkey-patch workarounds, and then reading the debate over this here and elsewhere, before finally deciding "eeeeh, I guess I should just conform to the spec published by the language/ecosystem I'm working with."Would it be a reasonable compromise for a spec like this one to relax the constraint to make things like those punctuation bits in pre-release identifiers optional, or, barring that, at least throw a call-out in the document itself noting something along the lines of "hey, we realize some specs like PEP-440 aren't quite compatible with semver as-is, but you can always adapt it to fit your needs and tools" as a warning to future newbies? Understanding that the goal of semver isn't to be something Python-specific, but also allowing for the fact that there are folks like myself who are using popular languages/platforms where it can cause problems, but who also appreciate and want to make use of the work done here with minimal fuss.
@holdenweb commented on GitHub (Jul 25, 2020):
I suspect the real reason for forbidding leading zeroes was to ensure release numbers had an unambiguous canonical form.