# Models
# cocoSsd
cocoSsd({ base?: string }): CocoSsd;
Object detection model based on tensorflow's COCO-SSD (opens new window) implementation. The model localizes and identifies multiple objects in a single image.
# Parameters
Option | Type | Description | Required | Default |
---|---|---|---|---|
base | string | Controls the base cnn model, can be 'mobilenet_v1', 'mobilenet_v2' or 'lite_mobilenet_v2'. lite_mobilenet_v2 is smallest in size, and fastest in inference speed. mobilenet_v2 has the highest classification accuracy. | 'lite_mobilenet_v2' |
# Streams
Name | Type | Description | Hold |
---|---|---|---|
$loading | Stream<boolean> | Defines if the model is loading | ✓ |
# Methods
# .predict()
async predict(img: ImageData): Promise<ObjectDetectorResults>
Make a prediction from an input image in ImageData
format. The method is asynchronous and returns a promise that resolves with the results of the prediction. The results have the following signature:
interface ObjectDetectorResults {
outputs: {
bbox: [number, number, number, number];
class: string;
confidence: number;
}[];
}
# Example
const source = marcelle.imageUpload();
const cocoClassifier = marcelle.cocoSsd();
const cocoPredictionStream = source.$images
.map(async (img) => cocoClassifier.predict(img))
.awaitPromises();
# knnClassifier
marcelle.knnClassifier({ k?: number, dataStore: DataStore }): KNNClassifier;
A K-Nearest Neighbors classifier based on Tensorflow.js's implementation (opens new window).
# Parameters
Option | Type | Description | Required | default |
---|---|---|---|---|
k | number | The K value to use in K-nearest neighbors. The algorithm will first find the K nearest examples from those it was previously shown, and then choose the class that appears the most as the final prediction for the input example. Defaults to 3. If examples < k, k = examples. | 3 | |
dataStore | DataStore | The dataStore used to store the model. This parameter is optional. |
The set of reactive parameters has the following signature:
parameters: {
k: Stream<number>;
}
# Streams
Name | Type | Description | Hold |
---|---|---|---|
$training | Stream<TrainingStatus> | Stream of training status events (see above), with no additional data |
# Methods
# .clear()
clear(): void
Clear the model, removing all instances
# .predict()
async predict(x: number[][]): Promise<ClassifierResults>
Make a prediction from an input feature array x
. The method is asynchronous and returns a promise that resolves with the results of the prediction. The results have the following signature:
interface ClassifierResults {
label: string;
confidences: { [key: string]: number };
}
# .train()
train(dataset: Dataset): void
Train the model from a given dataset.
# Example
const classifier = marcelle.knnClassifier({ k: 5 });
classifier.train(trainingSet);
const predictionStream = $featureStream // A stream of input features
.map(async (features) => classifier.predict(features))
.awaitPromises();
# mlpClassifier
marcelle.mlpClassifier({
layers?: number[],
epochs?: number,
batchSize?: number,
dataStore: DataStore
}): MLPClassifier;
A Multi-Layer Perceptron using Tensorflow.js. The configuration of the model (number of layers and number of hidden nodes per layer) can be configured.
# Parameters
Option | Type | Description | Required | Default |
---|---|---|---|---|
layers | number[] | The model configuration as an array of numbers, where each element defines a layer with the given number of hidden nodes | [64, 32] | |
epochs | number | Number of epochs used for training | 20 | |
batchSize | number | Training data batch size | 8 | |
dataStore | DataStore | The dataStore used to store the model. This parameter is optional. By default, a data store in memory will be created. |
The set of reactive parameters has the following signature:
parameters {
layers: Stream<number[]>;
epochs: Stream<number>;
batchSize: Stream<number>;
}
# Streams
Name | Type | Description | Hold |
---|---|---|---|
$training | Stream<TrainingStatus> | Stream of training status events, containing the current status ('idle' / 'start' / 'epoch' / 'success' / 'error'), the current epoch and associated data (such as loss and accuracy) during the training |
# Methods
# .clear()
clear(): void
Clear the model, removing all instances
# .predict()
async predict(x: number[][]): Promise<MLPResults>
Make a prediction from an input feature array x
. The method is asynchronous and returns a promise that resolves with the results of the prediction. The results have the following signature:
interface MLPResults {
label: string;
confidences: { [key: string]: number };
}
# .train()
train(dataset: Dataset): void
Train the model from a given dataset.
# Example
const classifier = marcelle.mlpClassifier({ layers: [64, 32], epochs: 50 });
classifier.train(trainingSet);
const predictionStream = $featureStream // A stream of input features
.map(async (features) => classifier.predict(features));
.awaitPromises();
# mobileNet
marcelle.mobileNet({
version?: 1 | 2,
alpha?: 0.25 | 0.50 | 0.75 | 1.0,
}): MobileNet;
The mobileNet component can be used both as a classification model and as a feature extractor. It is based on Tensorflow.js's Mobilenet implementation (opens new window). For feature extraction, the .process()
method can be used to get the embeddings from an input image.
# Parameters
Option | Type | Description | Required |
---|---|---|---|
version | 1 | 2 | The MobileNet version number. Use 1 for MobileNetV1 (opens new window), and 2 for MobileNetV2 (opens new window). Defaults to 1. | |
alpha | 0.25 | 0.50 | 0.75 | 1.0 | Controls the width of the network, trading accuracy for performance. A smaller alpha decreases accuracy and increases performance. 0.25 is only available for V1. Defaults to 1.0. |
Since parameters are used to load a heavy model, they can only be used on when the component is created, and there are not reactive parameters.
# Methods
# .predict()
async predict(image: ImageData): Promise<MobilenetResults>
Make a prediction from an input image image
in ImageData format. The method is asynchronous and returns a promise that resolves with the results of the prediction. The results have the following signature:
interface MobilenetResults {
label: string;
confidences: { [key: string]: number };
}
# .process()
async process(image: ImageData): Promise<number[][]>
Use mobilenet for feature extraction, for example to perform transfer learning. The method returns the embedding for the input image. The size of the embedding depends on the alpha (width) of the model.
# Example
const input = marcelle.webcam();
const m = marcelle.mobileNet();
// Extract features (embedding) from webcam images
const $embedding = input.$images.map((img) => m.process(img)).awaitPromises();
// Predict labels from webcam images (default mobilenet classification)
const $prediction = input.$images.map((img) => m.predict(img)).awaitPromises();
# onnxModel
onnxModel({
inputType: 'image' | 'generic';
taskType: 'classification' | 'generic';
segmentationOptions?: {
output?: 'image' | 'tensor';
inputShape: number[];
};
}): OnnxModel;
This component allows to make predictions using pre-trained models in the ONNX format, using onnxruntime-web
(opens new window). The default backend for inference is wasm
, as it provides a wider operator support.
The implementation currently supports tensors as input, formatted as nested number arrays, and two types of task (classification, generic prediction). Pre-trained models can be loaded either by URL, or through file upload, for instance using the fileUpload
component.
Such generic models cannot be trained.
# Methods
# .loadFromFile()
async loadFromFile(file: File): Promise<void>
Load a pre-trained ONNX model from a *.onnx
file.
# .loadFromUrl()
async loadFromUrl(url: string): Promise<void>
Load a pre-trained ONNX model from a URL.
# .predict()
async predict(input: InputType): Promise<OutputType>
Make a prediction from an input instance, which type depends on the inputType
specified in the constructor. The method is asynchronous and returns a promise that resolves with the results of the prediction.
Input types can be:
ImageData
if the model was instanciated withinputType: 'image'
TensorLike
(= array) if the model was instanciated withinputType: 'generic'
Output types can be:
ClassifierResults
if the model was instanciated withtaskType: 'classification'
TensorLike
if the model was instanciated withtaskType: 'generic'
Where classifier results have the following interface:
interface ClassifierResults {
label: string;
confidences: { [key: string]: number };
}
# Example
const source = imageUpload();
const classifier = tfjsModel({
inputType: 'image',
taskType: 'classification',
});
classifier.loadFromUrl();
const predictionStream = source.$images.map(async (img) => classifier.predict(img)).awaitPromises();
# tfjsModel
tfjsModel({
inputType: 'image' | 'generic';
taskType: 'classification' | 'segmentation' | 'generic';
segmentationOptions?: {
output?: 'image' | 'tensor';
applyArgmax?: boolean;
};
}): TFJSGenericModel;
This component allows to make predictions using pre-trained Tensorflow.js models, in either LayersModel (opens new window) or GraphModel (opens new window) format. This component supports:
- Models created with the tf.layers.*, tf.sequential(), and tf.model() APIs of TensorFlow.js and later saved with the tf.LayersModel.save() method.
- Models converted from Keras or TensorFlow using the tensorflowjs_converter.
It supports several types of input (currently, images or arrays), as well as several types of task (classification, segmentation, generic prediction). Pre-trained models can be loaded either by URL, or through file upload, for instance using the fileUpload
component.
Such generic models cannot be trained.
TIP
Note that exporting models from Keras to TFJS can lead to loading errors when particular layers are not implemented in TFJS. In this case, it is possible to export the Keras model to a saved_model and convert it to TFJS, where compatibility should be better.
# Methods
# .loadFromFiles()
async loadFromFiles(files: File[]): Promise<void>
Load a pre-trained TFJS model from a list files, that should include:
- a
model.json
file defining the model artifacts - one or several
.bin
weight files
# .loadFromUrl()
async loadFromUrl(url: string): Promise<void>
Load a pre-trained TFJS model from a URL.
# .predict()
async predict(input: InputType): Promise<OutputType>
Make a prediction from an input instance, which type depends on the inputType
specified in the constructor. The method is asynchronous and returns a promise that resolves with the results of the prediction.
Input types can be:
ImageData
if the model was instanciated withinputType: 'image'
TensorLike
(= array) if the model was instanciated withinputType: 'generic'
Output types can be:
ClassifierResults
if the model was instanciated withtaskType: 'classification'
ImageData | TensorLike
if the model was instanciated withtaskType: 'segmentation'
TensorLike
if the model was instanciated withtaskType: 'generic'
Where classifier results have the following interface:
interface ClassifierResults {
label: string;
confidences: { [key: string]: number };
}
# Example
const source = imageUpload();
const classifier = tfjsModel({
inputType: 'image',
taskType: 'classification',
});
classifier.loadFromUrl();
const predictionStream = source.$images.map(async (img) => classifier.predict(img)).awaitPromises();