mirror of
https://repo.dactyloidae.xyz/Dactyloidae/UXP.git
synced 2026-10-15 19:57:29 +09:00
import FIREFOX_52_6_0esr_RELEASE from mozilla-esr52 hg repo
This commit is contained in:
Roy Tam
commit
dcd9973243
150858 changed files with 23884658 additions and 0 deletions
381
testing/web-platform/tests/annotation-model/CONTRIBUTING.md
Normal file
381
testing/web-platform/tests/annotation-model/CONTRIBUTING.md
Normal file
|
|
@ -0,0 +1,381 @@
|
|||
Annotation-model: Guidelines for Contributing Tests
|
||||
===================================================
|
||||
|
||||
This document describes the method people should use for authoring tests and
|
||||
integrating them into the repository. Anyone is welcome to submit new tests to
|
||||
this collection. If you do, please create the tests following the guidelines
|
||||
below. Then submit them as a pull request so they can be evaluated
|
||||
|
||||
Structure
|
||||
---------
|
||||
|
||||
Tests are organized by major section of the Annotation Model specification. The
|
||||
folders associated with these are:
|
||||
|
||||
* annotations
|
||||
* bodiesTargets
|
||||
* collections
|
||||
* specificResources
|
||||
* selectors
|
||||
* states
|
||||
|
||||
Within these folders, special files ending with the suffix ".test" provide the source
|
||||
for the test as a set of declarative assertions about the required shape of the conforming
|
||||
JSON object. These files are transformed using a test generation tool into ".html" files
|
||||
that are then accessed by the Web Platform Test framework.
|
||||
|
||||
There are a few other folders that provide supporting materials for the tests:
|
||||
|
||||
* common - assertionObjects, conditionObjects, and other supporting materials
|
||||
* definitions - JSON Schema definitions that can be referenced
|
||||
* scripts - JavaScript that are included by tests
|
||||
* tools - supporting scripts and files
|
||||
|
||||
NOTE: The files in the definitions folder are expected to be JSON Schema
|
||||
definitions - basically commonly used concepts that are referenced by other JSON
|
||||
Schema files in the system. All of these 'definitions' are preloaded by the
|
||||
system before any other parts of a test are processed.
|
||||
|
||||
Test Cases
|
||||
----------
|
||||
|
||||
Each test is expressed as a simple (or complex) requirement in a test file.
|
||||
For each section of the document, the requirement is represented as a structure
|
||||
that describes the nature of the test, and then includes or references minimal
|
||||
JSON Schema that test the assertions implied by the requirement.
|
||||
|
||||
The structure of a test case is defined using a [JSON-LD
|
||||
Context](JSONtest-v1.jsonld). That context defines the following terms:
|
||||
|
||||
|Keyword | Values | Meaning
|
||||
|---------------|-----------------|---------
|
||||
|name | string | The name of this test for display purposes
|
||||
|description | string | A long self-describing paragraph that explains the purpose of the test and the expected input
|
||||
|ref | URI | An optional reference to the portion of the specification to which the test relates
|
||||
|testType | `automated`, `manual`, `ref` | The type of test - this informs [WPT](https://github.com/w3c/web-platform-tests) how the test should be controlled and presented
|
||||
|skipFailures | list of strings | An optional list of assertionType values that, if present, should have their test skipped if the result would be "unexpected". Defaults to the empty list.
|
||||
|assertions | list of URI, List @@@ATRISK@@@, or AssertionObject | The ordered collection of tests the input should be run against. See [JSON Schema Usage](#jsonSchema) for the structure of the objects. URI is relative to the top level folder of the test collection if it has a slash; relative to the current directory if it does not. @@@@ATRISK@@@@ Lists can be nested to define groups of sub-tests. Assertions / groups can be conditionally skipped. See [Assertion Lists](#assertionLists) for more details.
|
||||
|content | URI or object | An object containing content to be checked against the referenced assertions, or a URI from which to retrieve that content
|
||||
|
||||
Each test case has a suffix of `.test` and a shape like:
|
||||
|
||||
<pre>
|
||||
{
|
||||
"@context": "https://www.w3.org/ns/JSONtest-v1.jsonld",
|
||||
"name": "Verify annotation conforms to the model",
|
||||
"description": "Supply an example annotation that conforms to the basic structure.",
|
||||
"ref": "https://www.w3.org/TR/annotation-model/#model",
|
||||
"testType": "manual",
|
||||
"assertions": [
|
||||
"common/has_context.json",
|
||||
"common/has_id.json",
|
||||
{
|
||||
"$schema": "http://json-schema.org/draft-04/schema#",
|
||||
"title": "Verify annotation has target",
|
||||
"assertionType": "must",
|
||||
"expectedResult": "valid",
|
||||
"errorMessage": "The object was missing a required 'target' property",
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"target": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "array",
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "string"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"not": {"type": "object"}
|
||||
}
|
||||
},
|
||||
"required": ["target"]
|
||||
}
|
||||
]
|
||||
}
|
||||
</pre>
|
||||
|
||||
External references are used when the "assertion" is a common one that needs to
|
||||
be checked on many different test cases (e.g., that there is an @context in the
|
||||
supplied annotation).
|
||||
|
||||
NOTE: The title property of an assertionObject can contain markdown. This can
|
||||
help improve readability of the rendered assertions and debugging output.
|
||||
|
||||
NOTE: The content property does not yet have a defined use. One potential use would
|
||||
be to act as a pointer to a URI that can supply annotations from an implementation.
|
||||
In that case the URI would take a parameter with the test name as a way of telling
|
||||
the end point what test is running so it can deliver the right content.
|
||||
|
||||
### <a id="assertionLists">Assertion Lists</a> ###
|
||||
|
||||
The `assertion` list is an ordered list of assertions that will be evaluated
|
||||
against the submitted content. The list is *required*, and MUST have at least
|
||||
one entry. Entries in the list have the following types:
|
||||
|
||||
* AssertionObject
|
||||
|
||||
An in-line Object as defined in the section [Assertion
|
||||
Objects](#assertionObjects).
|
||||
* URI
|
||||
|
||||
A relative or absolute URI that references a AssertionObject in a .json file.
|
||||
If the URI is relative but contains no slashes, then it is considered to be
|
||||
in the current directory. If the URI is relative, contains slashes, but
|
||||
**does not start with a slash** then it is considered relative to the top of
|
||||
the tree of the current test collection (e.g., `annotation-model`).
|
||||
* List @@@ATRISK@@@
|
||||
|
||||
A nested Assertion List. While nested Assertion Lists are optional, if one
|
||||
is present it MUST have at least one entry. Entries are as in this list.
|
||||
Assertion Lists can be nested to any depth (but don't do that - it would be
|
||||
too hard to maintain).
|
||||
|
||||
|
||||
<a id="assertionObjects">Assertion Objects</a>
|
||||
-----------------
|
||||
|
||||
In this collection of tests, Assertion Objects can be contained inline in the
|
||||
`.test` files or contained in external files with the suffix `.json`. The
|
||||
vocabularly and structure is as defined in [JSON Schema
|
||||
v4](http://json-schema.org/documentation.html) augmented with some additional
|
||||
properties defined in this section.
|
||||
|
||||
In general each JSON Schema definition provided in this test suite should be as
|
||||
minimal as possible. This promotes clarity and increases the likelihood that
|
||||
it is correct. While it is ---possible--- to create JSON Schema files that
|
||||
enforce many different requirements on a data model, it increases the
|
||||
complexity and can also reduce the atomicity of tests / sub-tests (because a
|
||||
test ends up testing more than one thing). Please try to avoid creating
|
||||
complex JSON Schema. (A notable exception is the situation where multiple
|
||||
properties of a structure are interdependent.)
|
||||
|
||||
Tools such as [the JSON Schema Creator](http://jsonschema.net/) may be helpful
|
||||
in creating schema snippets that can be integrated into JSONtest Assertion
|
||||
Objects. Remember that the JSON Schema you create should be as permissive as
|
||||
possible to address the various shapes that a give property might take (e.g., a
|
||||
'foo' might be a string or an object that contains sub-properties that express
|
||||
the string, or an array of 1 or more objects with those sub-properties).
|
||||
|
||||
In addition to the validation keys defined in JSON Schema v4, Schema files in
|
||||
this collection are also permitted to use the following keywords:
|
||||
|
||||
|Keyword | Values | Meaning |
|
||||
|---------------|-----------------|---------|
|
||||
|onUnexpectedResult | `failAndContinue`, `failAndSkip`, `failAndAbort`, `passAndContinue`, `passAndSkip`, `passAndAbort` | Action to take when the result is not as expected. Default is `failAndContinue` |
|
||||
|assertionType | `must`, `may`, `should` | Informs the system about the severity of a failure. The default is `must` |
|
||||
|assertionFile | URI | An external file that contains an assertion SCHEMA. When this value is supplied, and local properties will override the ones loaded from the external file.
|
||||
|errorMessage | string | A human readable explanation of what it means if the test fails. |
|
||||
|expectedResult | `valid`, `invalid` | Tells the framework whether validating against this schema is expected to succeed or fail. The default is `valid` |
|
||||
|
||||
|
||||
### Example Assertion Object ###
|
||||
|
||||
<pre>
|
||||
{
|
||||
"$schema": "http://json-schema.org/draft-04/schema#",
|
||||
"title": "Verify annotation has @context",
|
||||
"type": "object",
|
||||
"expectedResult" : "valid",
|
||||
"properties": {
|
||||
"@context": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "string"
|
||||
},
|
||||
{
|
||||
"type": "array",
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "string"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"not": {"type": "object"}
|
||||
}
|
||||
},
|
||||
"required": ["@context"]
|
||||
}
|
||||
</pre>
|
||||
|
||||
Note that in the case where a feature is *optional* the JSON Schema MUST be
|
||||
crafted such that if the attribute is permitted to be missing from the content
|
||||
(so that the result is `true`), but when the attribute is present in the
|
||||
content it conforms to any requirements.
|
||||
|
||||
|
||||
|
||||
<a id="conditionObjects">Condition Objects</a>
|
||||
-----------------
|
||||
|
||||
A Condition Object is a sub-class of an Assertion Object. It allows the
|
||||
specification of the evaluation strategy for the assertions referenced by the
|
||||
object. An object is a Condition Object IFF it has a `assertions` property. In
|
||||
this case, there MUST NOT be an `assertionFile` property.
|
||||
|
||||
|
||||
|Keyword | Values | Meaning |
|
||||
|---------------|-----------------|---------|
|
||||
|compareWith | `and`, `or` | How should the result of any referenced assertions be compared. Defaults to `and`. Note that this implies there is also an assertions property with a nested list of assertions to compare. |
|
||||
|assertions | a list of assertions as in a Test Case above. This is required if there is a compareWith property |
|
||||
|
||||
|
||||
An example of a test that would pass if there were an `@context` OR there were an `@id`:
|
||||
|
||||
<pre>
|
||||
{
|
||||
"@context": "https://www.w3.org/ns/JSONtest-v1.jsonld",
|
||||
"name": "A test that has an 'or' clause",
|
||||
"description": "A complex test that uses or-ing among a list of assertions",
|
||||
"ref": "https://www.w3.org/TR/annotation-model/#model",
|
||||
"testType": "manual",
|
||||
"assertions": [
|
||||
{ "$schema": "http://json-schema.org/draft-04/schema#",
|
||||
"title": "must have context or id",
|
||||
"description": "A more complex example that allows one of many options to pass",
|
||||
"assertions": [
|
||||
{ "title": "Condition Object",
|
||||
"description": "A pseudo-test that will get a result from the aggregate of its children",
|
||||
"assertionType": "must",
|
||||
"expectedResult": "valid",
|
||||
"errorMessage": "Error: None of the various options were present",
|
||||
"compareWith": "or",
|
||||
"assertions": [
|
||||
"common/has_context.json",
|
||||
"common/has_id.json"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
</pre>
|
||||
|
||||
|
||||
Command Line Tools
|
||||
------------------
|
||||
|
||||
### Building the Test Files ###
|
||||
|
||||
The actual .html test case files are generated using the script
|
||||
tools/make_tests.py. This script will search the directory heriarchy looking for
|
||||
files ending on `.test` and creating `.html` files from them using the template in
|
||||
the tools folder. If you want to regenerate the examples too, supply the
|
||||
`--examples` option to the script.
|
||||
|
||||
Note that when submitting tests to the repository, the `.html` versions must be
|
||||
included.
|
||||
|
||||
### Testing the Tests ###
|
||||
|
||||
### Driving Tests with Input Files ###
|
||||
|
||||
Complex Examples
|
||||
----------------
|
||||
|
||||
This section is a collection of more complex examples to illustrate the
|
||||
expressive power of the [Assertion List](#assertionLists) structure. These can
|
||||
be used as templates for creating actual `.test` files.
|
||||
|
||||
### Including and Overriding an Assertion ###
|
||||
|
||||
Assertions can be contained in external `.json` files. It is possible for an
|
||||
object in an Assertion List to include the external file and override one or
|
||||
more of its properties:
|
||||
|
||||
<pre>
|
||||
{
|
||||
"@context": "https://www.w3.org/ns/JSONtest-v1.jsonld",
|
||||
"name": "Permit no target property",
|
||||
"description": "Ensure there is no 'target' property when there is a 'randomName' property in the Annotation",
|
||||
"assertions": [
|
||||
{
|
||||
"$schema": "http://json-schema.org/draft-04/schema#",
|
||||
"title": "Verify annotation has randomName",
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"randomName": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"required": ["randomName"]
|
||||
},
|
||||
{ "assertionFile" : "common/target.json",
|
||||
"title" : "Require target to be missing",
|
||||
"expectedResult" : "invalid",
|
||||
"errorMessage" : "The target MUST not be present when 'randomName' is also present",
|
||||
}
|
||||
]
|
||||
}
|
||||
</pre>
|
||||
|
||||
### Nested Assertion Collections with Skip ###
|
||||
|
||||
Assertion Lists can be nested within Assertion Lists. This feature, combined
|
||||
with the `onUnexpectedResult` property, makes it possible to skip a collection
|
||||
of tests when an assertion in the list is not satisfied:
|
||||
|
||||
<pre>
|
||||
{
|
||||
"@context": "https://www.w3.org/ns/JSONtest-v1.jsonld",
|
||||
"name": "If there is no 'target' property, skip some tests",
|
||||
"description": "When 'target' is not present, other properties related to 'target' are not required",
|
||||
"assertions": [
|
||||
"common/context.json",
|
||||
[
|
||||
{ "assertionFile" : "common/target.json",
|
||||
"errorMessage" : "Target was not present so skip the rest of this section",
|
||||
"onUnexpectedResult" : "failAndSkip"
|
||||
},
|
||||
"sometest.json",
|
||||
"sometest2.json",
|
||||
"sometest3.json"
|
||||
]
|
||||
]
|
||||
} ;
|
||||
</pre>
|
||||
|
||||
### Assertion that finds a specific @context Value ###
|
||||
|
||||
Sometimes you want a property to be flexible, but to have one and only one of a
|
||||
specific value. This is especially true with, for example, @context in JSON-LD.
|
||||
One way you might do this is:
|
||||
|
||||
<pre>
|
||||
{
|
||||
"$schema": "http://json-schema.org/draft-04/schema#",
|
||||
"title": "Verify a specific @context",
|
||||
"type": "object",
|
||||
"expectedResult" : "valid",
|
||||
"properties": {
|
||||
"@context": {
|
||||
"anyOf": [
|
||||
{
|
||||
"type": "string"
|
||||
"enum": [ "http://www.w3.org/ns/anno.jsonld" ]
|
||||
},
|
||||
{
|
||||
"type": "array",
|
||||
"minitems": "1",
|
||||
"uniqueItems": true,
|
||||
"additionalItems": true,
|
||||
"items" : [
|
||||
{ "type": "string",
|
||||
"enum": [ "http://www.w3.org/ns/anno.jsonld" ]
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"not": {"type": "object"}
|
||||
}
|
||||
},
|
||||
"required": ["@context"]
|
||||
}
|
||||
|
||||
</pre>
|
||||
Loading…
Add table
Add a link
Reference in a new issue