[GH-ISSUE #718] Include examples of tests more comprehensive than those in the main document #1355

New Issue

GiteaMirror · 2026-04-16T10:45:00-05:00

GiteaMirror commented

2026-04-16 10:45:00 -05:00

Originally created by @krader1961 on GitHub (Jun 28, 2021).
Original GitHub issue: https://github.com/semver/semver/issues/718

I was working on an implementation for the Elvish shell to compare two semantic versions and was dismayed that this project did not have a comprehensive set of tests I could use to verify my implementation. It would be really helpful if there was a file that included two semver strings and the expected zero, -1, or 1 result of comparing the two strings. This, obviously, could be as simple as a space/comma/tab delimited file such as this:

-1 1.0.0-alpha 1.0.0
0 0.1.0+build 0.1.0
1 0.10.3 0.3.4

Or, it could be something with more structure, such as JSON. Either way, it would be really helpful to writers of semver comparison functions if there was a sanctioned set of unit tests that covered all the corner cases.

Originally created by @krader1961 on GitHub (Jun 28, 2021). Original GitHub issue: https://github.com/semver/semver/issues/718 I was working on an implementation for the [Elvish shell](https://elv.sh) to compare two semantic versions and was dismayed that this project did not have a comprehensive set of tests I could use to verify my implementation. It would be really helpful if there was a file that included two semver strings and the expected zero, -1, or 1 result of comparing the two strings. This, obviously, could be as simple as a space/comma/tab delimited file such as this: ``` -1 1.0.0-alpha 1.0.0 0 0.1.0+build 0.1.0 1 0.10.3 0.3.4 ``` Or, it could be something with more structure, such as JSON. Either way, it would be really helpful to writers of semver comparison functions if there was a sanctioned set of unit tests that covered all the corner cases.

GiteaMirror commented

2026-04-16 10:45:01 -05:00

@ljharb commented on GitHub (Jun 28, 2021):

Try https://npmjs.com/semver

@ljharb commented on GitHub (Jun 28, 2021): Try https://npmjs.com/semver

GiteaMirror commented

2026-04-16 10:45:02 -05:00

@krader1961 commented on GitHub (Jun 28, 2021):

@ljharb, Thanks for that link but it does not address my concern. That a specific implementation includes relevant unit tests does not address my concern. The unit tests in that project should be exposed directly by this project.

@krader1961 commented on GitHub (Jun 28, 2021): @ljharb, Thanks for that link but it does not address my concern. That a specific implementation includes relevant unit tests does not address my concern. The unit tests in that project should be exposed directly by this project.

GiteaMirror commented

2026-04-16 10:45:02 -05:00

@krader1961 commented on GitHub (Jun 30, 2021):

@ljharb, Thanks for the link but it doesn't address my request. It looks like the tests are very npmjs.com specific; such as https://github.com/npm/node-semver/blob/master/tap-snapshots/test-bin-semver.js-TAP.test.js. What would be useful is to extract the basic comparison tests from that project into a language agnostic data file that is part of this project. In issue #717 someone said you're very knowledgeable about semantic versioning which implies you would be someone who would be able to add such a data set to this project for the benefit of any project that wants to implement support for semver.

@krader1961 commented on GitHub (Jun 30, 2021): @ljharb, Thanks for the link but it doesn't address my request. It looks like the tests are very npmjs.com specific; such as https://github.com/npm/node-semver/blob/master/tap-snapshots/test-bin-semver.js-TAP.test.js. What would be useful is to extract the basic comparison tests from that project into a language agnostic data file that is part of this project. In issue #717 someone said you're very knowledgeable about semantic versioning which implies you would be someone who would be able to add such a data set to this project for the benefit of any project that wants to implement support for semver.

GiteaMirror commented

2026-04-16 10:45:03 -05:00

@ljharb commented on GitHub (Jun 30, 2021):

I appreciate the vote of confidence, but I’m not sure i understand what you’re looking for. The package i linked to has many APIs that accept ranges (which, until #584, is npm-specific) but it also has APIs which validate semver versions, afaik according to this specification.

@ljharb commented on GitHub (Jun 30, 2021): I appreciate the vote of confidence, but I’m not sure i understand what you’re looking for. The package i linked to has many APIs that accept ranges (which, until #584, is npm-specific) but it also has APIs which validate semver versions, afaik according to this specification.

GiteaMirror commented

2026-04-16 10:45:03 -05:00

@krader1961 commented on GitHub (Jun 30, 2021):

I appreciate the vote of confidence, but I’m not sure i understand what you’re looking for.

See the first comment in this issue. This project should have a data set comprised of two semvers and a value indicating whether they are semantically equal or the first semver is older or newer than the second semver. That data set should be in a language agnostic format such as JSON or a simple CSV text file. Then any project, such as https://npmjs.com/semver, can use that data set to validate they are correctly comparing semvers.

@krader1961 commented on GitHub (Jun 30, 2021): > I appreciate the vote of confidence, but I’m not sure i understand what you’re looking for. See the first comment in this issue. This project should have a data set comprised of two semvers and a value indicating whether they are semantically equal or the first semver is older or newer than the second semver. That data set should be in a language agnostic format such as JSON or a simple CSV text file. Then any project, such as https://npmjs.com/semver, can use that data set to validate they are correctly comparing semvers.

GiteaMirror commented

2026-04-16 10:45:03 -05:00

@krader1961 commented on GitHub (Jun 30, 2021):

To be determined is how to encode invalid semvers in a test data set so that an implementation can reliably confirm they are correctly handling an invalid semver. I will note, however, that the data set should not include semvers with a "v" prefix since that is so common it needs to be ignored except by strictly conforming implementations.

@krader1961 commented on GitHub (Jun 30, 2021): To be determined is how to encode invalid semvers in a test data set so that an implementation can reliably confirm they are correctly handling an invalid semver. I will note, however, that the data set should not include semvers with a "v" prefix since that is so common it needs to be ignored except by strictly conforming implementations.

GiteaMirror commented

2026-04-16 10:45:04 -05:00

@krader1961 commented on GitHub (Jul 1, 2021):

The data in issue #567 should be leveraged when resolving this issue.

@krader1961 commented on GitHub (Jul 1, 2021): The data in issue #567 should be leveraged when resolving this issue.

GiteaMirror commented

2026-04-16 10:45:04 -05:00

@jwdonahue commented on GitHub (Nov 29, 2021):

@krader1961, I developed a SemVer oracle generator a while back. I have the C# code, probably targets .NET Core 3.1, but I can dust it off and upgrade it to .NET 5 pretty easily. If you're interested, I'll publish it and we can work on shaping a set of oracles to fit any statistical profile. I never did get more data for #567 and I returned to work in Washington state shortly after we finally got a better internet connection.

I think I parked the project because I didn't have a way to independently validate the correctness of the generated oracles. I am busy working a contract these days, but I can spare a few cycles on the week-ends/evenings to further this along, if I can count on some feed back and assistance.

See my profile for contact info.

@jwdonahue commented on GitHub (Nov 29, 2021): @krader1961, I developed a SemVer oracle generator a while back. I have the C# code, probably targets .NET Core 3.1, but I can dust it off and upgrade it to .NET 5 pretty easily. If you're interested, I'll publish it and we can work on shaping a set of oracles to fit any statistical profile. I never did get more data for #567 and I returned to work in Washington state shortly after we finally got a better internet connection. I think I parked the project because I didn't have a way to independently validate the correctness of the generated oracles. I am busy working a contract these days, but I can spare a few cycles on the week-ends/evenings to further this along, if I can count on some feed back and assistance. See my profile for contact info.

GiteaMirror commented

2026-04-16 10:45:05 -05:00

@jwdonahue commented on GitHub (Nov 29, 2021):

@krader1961, the regex FAQ has links to regex101.com that include some minimal test oracles that are aimed at testing regexes.

@jwdonahue commented on GitHub (Nov 29, 2021): @krader1961, the [regex FAQ](https://semver.org/#is-there-a-suggested-regular-expression-regex-to-check-a-semver-string) has links to regex101.com that include some minimal test oracles that are aimed at testing regexes.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: github-starred/semver#1355