brainome logo

303 Predictor Validation Measurements in JSON

The predictor can output validation measurements in json rather than human-readable text.

  • Validation measurements in json format.

Prerequisites

This notebook assumes brainome is installed as per notebook brainome_101_Quick_Start

The data sets are:

!python3 -m pip install brainome --quiet
!brainome --version

import urllib.request as request
response2 = request.urlretrieve('https://download.brainome.ai/data/public/titanic_validate.csv', 'titanic_validate.csv')
%ls -lh titanic_validate.csv
brainome v1.006-19-prod
-rw-r--r-- 1 runner docker 5.8K Oct 20 18:09 titanic_validate.csv

Generate a predictor

The predictor filename is predictor_303.py

!brainome https://download.brainome.ai/data/public/titanic_train.csv -y -o predictor_303.py -modelonly -q
print('\nCreated predictor_303.py')
!ls -lh predictor_303.py
WARNING: Could not detect a GPU. Neural Network generation will be slow.
Created predictor_303.py
-rw-r--r-- 1 runner docker 39K Oct 20 18:09 predictor_303.py

Validation measurements in json format.

The same measurements as all previous exercises can be generated in JSON format for further system integration Use -validate -json to trigger the predictor to output json validation measurements.

!python3 predictor_303.py -validate titanic_validate.csv -json > validation_measurements_303.json
import json
with open('validation_measurements_303.json', 'r') as measurement_file:
    validation_measurements = json.load(measurement_file)
    print(json.dumps(validation_measurements, indent=4))
{
    "instance_count": 80,
    "classifier_type": "RF",
    "classes": 2,
    "number_correct": 65,
    "accuracy": {
        "best_guess": 61.25,
        "improvement": 20.0,
        "model_accuracy": 81.25
    },
    "false_negative_instances": 10,
    "false_positive_instances": 5,
    "true_positive_instances": 21,
    "true_negative_instances": 44,
    "false_negatives": 12.5,
    "false_positives": 6.25,
    "true_negatives": 55.0,
    "true_positives": 26.25,
    "model_capacity": 41,
    "generalization_ratio": 1.5218042472451754,
    "model_efficiency": 0.48,
    "shannon_entropy_of_labels": 0.9631672450918831,
    "classbalance": [
        0.6125,
        0.3875
    ],
    "confusion_matrix": [
        [
            44,
            5
        ],
        [
            10,
            21
        ]
    ],
    "multiclass_stats": {
        "0": {
            "TP": 44,
            "FP": 10,
            "FN": 5,
            "TN": 21
        },
        "1": {
            "TP": 21,
            "FP": 5,
            "FN": 10,
            "TN": 44
        }
    }
}

Next Steps

TODO next…