Value Processing

Value processors are functions that are passed a value and are expected to either return a value, return null, return undefined, or throw an error. They can be synchronous or asynchronous, Returned Promises are handled automatically.

They are used inside handler pipelines, where each value processor's output becomes the input of the next processor in the pipeline.

Here are a bunch of handlers with pipelines that update strings as they pass through:

import { Schema, SchemaResolver, SchemaPolicy } from '@versionzero/schema';
const resolver = new SchemaResolver();
const schema = await resolver.compile(
  new Schema('string')
    .normalizer(v => `n1(${v})`)
    .normalizer(v => `n2(${v})`)
    .normalizer([v => `n3(${v})`, v => `n4(${v})`])
    .transformer(v => `t1(${v})`)
    .transformers(v => `t0(${v}`, SchemaPolicy.PREPEND)
    .transformer(v => `t2(${v})`)
    .finalizer(v => `f(${v})`)
    .option('revalidate', false)  
    .validator([v => `v1(${v})`, v => `v2(${v})`])
    .serializer(v => `s(${v})`)
);
console.log( await schema.process('ABC') );
// -> v2(v1(f(t2(t1(t0(n4(n3(n2(n1(ABC)))))))))
console.log( await schema.validate('ABC') );
// -> v2(v1(ABC))
console.log( await schema.serialize('ABC'));
// -> s(ABC)

(See note on revalidation) Notice how the pipeline can be built piecemeal with multiple fluent calls, or in bulk by passing arrays.

The various handlers operate at different phases of overall schema value processing and interpret results differently, but all value processor pipelines are compiled and executed in the same manner, allowing value processors to be "mixed and matched" as needed in any handler (although some value processors are better suited to specific handlers). All value processors have the same signature and return value semantics.

When a value processor returns null, it is interpreted as "explicitly omit this value from the output".

An undefined result is different. It indicates "no value can currently be produced", resulting in a "retry" - the schema being revisited on another traversal pass. Multi-pass traversal runs until the output is complete or stabilizes, enabling dynamic defaults, conditionals, unions, and complex cross-dependencies to resolve.

Value processors are often used to check input values, and signal exceptions by throwing errors. By default, errors will propagate out and be re-thrown by the calling handler, but they may be absorbed if the processor is wrapped inside another processor (e.g. $try or $gate).

By convention throughout the library, value processors that produce values as referenced as operators, and value processors that focus on checking values are referenced as constraints.

Value Processor Registry

The SchemaResolver provides a value processor registry, used to resolve value processors referenced by keyword during compilation. To help disambiguate keywords from ordinary strings, keywords are always prefixed with a dollar sign ($).

The registry is preloaded with an extensive collection of prebuilt value processors, but you are encouraged to create and share your own libraries of processors to load into the registry.

Value processor keywords are automatically retrieved from the registry during compilation.

Value Processor Compilation

The final compiled form of a value processor is a function, but they are parsed from the Schema handler in multiple supported formats:

• Inline function
• Keyword reference, either simple ($keyword) or parameterized ({$keyword: <args>})
• Containers of value processors (members are individually processed)
• Literal values (if necessary, disambiguate using {$literal: <value>})

import { Schema, SchemaResolver} from '@versionzero/schema';
const resolver = new SchemaResolver();

const schema = await resolver.compile(
  new Schema('object')
          .property('i', new Schema()
                  .transformer(v => `inline: ${v}`)
          )
          .property('k', new Schema()
                  .normalizer('$title-case')
                  .validator({$length: {min: 1}})
          )
          .property('a', new Schema()
                  .normalizer([['$kebab-case', '$pascal-case', '$constant-case']])
          )
          .property('o', new Schema()
                  .normalizer({k: '$kebab-case', p: '$pascal-case', c: '$constant-case'})
          )
          .property('c1', new Schema().normalizer(123))
          .property('c2', new Schema().normalizer('hi'))
          .property('c3', new Schema().normalizer(['x', 'y', 'z']))
          .property('l', new Schema()
                  .normalizer({$literal: {k: '$kebab-case', p: '$pascal-case', c: '$constant-case'}})
          )
);

// Let's see what we get when we assign the same value to
// all the properties...
const input = {};
for (const [property,] of schema.propertyEntries) {
  input[property] = 'hello world';
}

console.log( await schema.process(input) );
/* outputs...
{
  i: 'inline: hello world',
  k: 'Hello World',
  a: [ 'hello-world', 'HelloWorld', 'HELLO_WORLD' ],
  o: { k: 'hello-world', p: 'HelloWorld', c: 'HELLO_WORLD' },
  c1: 123,
  c2: 'hi',
  c3: 'z',
  l: { k: '$kebab-case', p: '$pascal-case', c: '$constant-case' }
}
 */

Writing Simple Value Processors

Here's a trivial operator:

import { Schema, SchemaResolver} from '@versionzero/schema';
const resolver = new SchemaResolver();

function toLowerCase(value) {
  return String(value).toLowerCase();
}
// use it directly...
new Schema('string').normalizer(toLowerCase);

// or register it, and reference by keyword...
resolver.registerValueProcessor('to-lower-case', toLowerCase);
new Schema('string').normalizer('$to-lower-case');

And here's a constraint:

function isEven(value) {
  if (value % 2 !== 0) {
    throw new Error(`${value} is not an even number`);
  }
  return value;
}
// as above, you can reference it directly...
new Schema('number').validator(isEven);

// or register it...
resolver.registerValueProcessor('is-even', isEven);
new Schema('number').validator('$is-even');

In both cases, these value processors only need to look at the input value.

Empty Signal

Any schema that allows child properties is required to normalize the symbol Schema.EMPTY as into an empty container of the appropriate type. The prebuilt object schema's normalizer is built using the $object processor, which will normalize Schema.EMPTY as {}. Similarly for array, the $array processor returns []. If you extend one of these prebuilt container schemas, your own normalizers will be "downstream" in the pipeline, and will receive these values, but if you build a container schema "from scratch", you must handle the symbol yourself.

(For consistency, the other prebuilt schemas interpret Schema.EMPTY with their own semantics, e.g. string returns "", number returns 0, and so forth. Support is required for containers, but schemas that don't support Schema.EMPTY should typically throw an exception if it is received during normalization.)

"Success" vs. "Defined" vs. "Truthiness"

The library semantics tend to focus on "return a value, or throw an error". This is easy to forget when writing validation logic:

const rx = /[a-z0-9]+/i;
// incorrect:
new Schema('string').validator(value => rx.test(value));  // oops! returns a boolean!

// correct:
new Schema('string').validator(value => {
  if (rx.test(value)) { 
    return value;                        // pass through on success 
  }
  throw new Error('not alphanumeric!')   // throw on failure
})

// much nicer to use prebuilts or your own registered keywords:
new Schema('string').validator('$alphanum');

Revalidation catches some accidental value changes, but not all.

The library provides a set of conditional processors that vary in what they inspect. $gate and $when care if values are defined. $try only cares if the processor threw an error or not. $if and $check evaluate whether a value is truthy.

Sequence processors similarly vary in behavior. $all just ensures all processors succeeded, whereas $and requires them to all return truthy values.

Truthiness

The library differs from JavaScript semantics in that defined string values are not all treated as a boolean true. The following string values are all treated (in a case-insensitive manner) as true: "true", "yes", "on", "enabled", "active", and "1". More importantly (since those all would have been interpreted as true anyway), the opposite string values are treated as false: "false", "no", "off", "disabled", "inactive", and "0".

These semantics are used by the boolean schema normalizer via the $boolean processor, as well as all conditional and sequence processors that check truthiness.

Parameterized Value Processors

Some value processors accept additional arguments. For example,

new Schema('number')
  .validator('$integer')
  .validator({$range: {min: 1, max: 6}})

The calling conventions for parameterized processors are as follows:

1. Referencing a $keyword as a string is the same as passing an empty argument array. Example: .normalizer('$round').
2. Arguments can be passed by ensuring the $keyword is the only key of an object, and providing either a single argument value, an argument array or an argument object. Examples:
   1. .normalizer({$round: 1}) (treated as if it were a single element array)
   2. .validator({$range: [1, 6]}
   3. .validator({$range: {min: 1, max: 6}}) or .normalizer({$round: {precision: 1}})
3. Parameterized processors usually interpret arrays by assigning each array value as an argument to the parameter by definition sequence. Some processors treat arrays as values, e.g. $pipeline or $all. See the individual processor documentation for details.
4. Arguments are usually fully interpreted as if they were value processors themselves, enabling complex evaluation. You can force an argument to be interpreted as a simple value using {$literal: <value>}.

Creating your own parameterized value processors is an advanced topic, touched on below.

Advanced Value Processors

The full signature of a value processor actually has several additional parameters that are useful in more complex situations:

/**
 * @param {any} value - input value to process
 * @param {any} target - root of full output being built
 * @param {SchemaLocation} location - cursor within traversal
 * @param {object} options - user options, traversal state, processor arguments
 * @returns {any|Promise<any>} - output value
 */
function process(value, target, location, options) { /* ... */ }

The value parameter is largely self-explanatory, as it is the input to the processor, and typically the main focus of what needs processing. As you will see, many processors only look at this argument.

The target parameter corresponds to the output value, not just of the current schema, but of the entire traversal. It can be inspected for "peer values" that have been written from other schemas, but there is no guarantee of ordering. A processor should generally not blindly assume that other peer schemas have successfully output a value to target; they should check for a value first, and return undefined if the dependency is not yet available.

The location parameter receives a SchemaLocation instance, which provides a reference both to the current schema, as well as the navigation path where the schema was found within the entire hierarchy. Schemas don't know about "parents", as a given instance may be linked in multiple locations, but the location can be used to navigate to the parent or peers. (The path also corresponds to the location in the target where the current schema will write its output.) See the documentation for SchemaLocation for details.

Finally, the options parameter allows callers of CompiledSchema.process() or CompiledSchema.validate() to pass extra information down to processors. For advanced value processing, it also contains the active traversal state, and an args member for value processors that define parameters.

Here's an example that leverages the target and location parameters:

import { Schema, SchemaResolver } from '@versionzero/schema';
const resolver = new SchemaResolver();

const getDeepValue = (obj, path) => {
  return path.split('.').reduce((acc, part) => acc?.[part], obj);
};


const schema = await resolver.compile(
  new Schema('object')
    .deep()
    .property('rounds', new Schema('number').default(1))
    .property('flips', new Schema('array')
        .default((value, target) => {
          // dynamic default, populate based on the "rounds" value
          if (target?.rounds === undefined) { return undefined }
          return Array.from({ length: target.rounds }, 
            () => ({coin: Math.random()> 0.5? 'heads' : 'tails'}) );
        })
        .property('*', new Schema('object')
          .property('coin', new Schema('string'))
          .property('winner', new Schema('string')
            .default('nobody')
            .transformer((value, target, location) => {
              // unlike "target.rounds" above, we don't know the absolute
              // path in the target of our peer "coin" property, so we have
              // to use the SchemaLocation to navigate:
              const coinLocation = location.relative('^.coin');
              const coinValue = getDeepValue(target, coinLocation.path);
    
              return (coinValue === undefined)
                     ? undefined              
                     : (coinValue === 'heads' ? 'me' : 'you');
            })
          )
        )
    )
    .property('result', new Schema('string')
      .default('nothing happens')
      .condition((value, target, location, options) => {
        // This isn't enough, because the array might exist but be partially populated:
        if (target?.flips.length !== target?.rounds) { return false };
        
        // We'd need to actually inefficiently probe to see if all winners are done:
        //if (target.flips.some(flip => flip.winner === undefined)) { return false }

        // Instead, we'll use an advanced technique of checking whether the
        // assignment as completed.  (This sort of thing is almost never
        // necessary to do by hand.)
        return options.context.getState('flips').completed;
      })
      .transformer((value, target) => {
        const scores = target.flips.reduce((acc, flip) => {
          acc[flip.winner]++;
          return acc;
        }, {me: 0, you: 0});
        
        if (scores.me > scores.you) {
          return 'You buy me ice cream';
        }
        else if (scores.you > scores.me) {
          return 'I buy you ice cream';
        }
        else {
          return 'no winner';
        }
        
      }
    )
    )
);
console.log( await schema.process({rounds:10}) );

The most common needs for referencing the extended parameters are provided via prebuilt value processors. Here's the same schema, but with less magic code:

import { Schema, SchemaResolver } from '@versionzero/schema';
const resolver = new SchemaResolver();

const schema = await resolver.compile(
  new Schema('object')
    .deep()
    .property('rounds', new Schema('number').default(1))
    .property('flips', new Schema('array')
        .default((value, target) => {
          // dynamic default, populate based on the "rounds" value
          if (target?.rounds === undefined) { return undefined }
          return Array.from({ length: target.rounds }, 
            () => ({coin: Math.random()> 0.5? 'heads' : 'tails'}) );
        })
        .property('*', new Schema('object')
          .property('coin', new Schema('string'))
          .property('winner', new Schema('string')
            .default('nobody')
            .transformer([{$relative: '^.coin'}, coinValue => (coinValue === 'heads' ? 'me' : 'you')])
          )
        )
    )
    .property('result', new Schema('string')
      .default('nothing happens')
      .condition({$reference: {path: '^.flips', completed: true}})
      .transformer((value, target) => {
        const scores = target.flips.reduce((acc, flip) => {
          acc[flip.winner]++;
          return acc;
        }, {me: 0, you: 0});
        
        if (scores.me > scores.you) {
          return 'You buy me ice cream';
        }
        else if (scores.you > scores.me) {
          return 'I buy you ice cream';
        }
        else {
          return 'no winner';
        }
        
      }
    )
    )
);
console.log( await schema.process({rounds:10}) );

Arguments are passed to value processors via options.args. You can implement your own logic for interpreting and enforcing calling conventions, but the parsing rules will pre-evaluate what is passed:

For example, in this call...

schema.validator({$whatever: {first: {$pipeline: [5, (v => v + 1)]}}})

...the $whatever processor receives options.args = {first: 6}.

whereas in this call (when transforming the value 3)...

schema.transformer({$something: ['$positive', '$negative']})

...the $something processor receives options.args = [true, false], as each array member is treated as a separate processor that receives the current input value.

More complex parameter processing can be achieved using the extended "processor builder" registration methodology (see the sources of many prebuilt processors), but this is an advanced topic subject to API churn, and not yet part of the public API.