HTM community detector (py3) by breznak · Pull Request #5 · htm-community/NAB

breznak · 2019-07-24T14:18:31Z

Adds htm.core https://github.com/htm-community/htm.core as a detector

Currently WIP, ~~please review after #2 is merged as this builds atop that.~~

Features

native htm.core improvements
python3
installs htm.core seamlessly
finish the HtmcoreDetector detector
- inspired from nab/detectors/numenta/numenta*_detector.py here; py/htm/examples/hotgym.py in htm.core
run on all data and commit results
- add to comparison Score board

TODO optional in separate PRs

use swarming to optimize parameters for the datasets Optimize HTM.core detector params #6

For numenta#337 where it should be then submitted as well

Fixes htm-community/htm.core#205

install htm.core detector from setup.py if installed, just skip, otherwise build from github

runable POC for community HTM detector from famous htm.core repo!

breznak

Review with planned work to do.

If you have time, @dkeeney @ctrl-z-9000-times please have a look at the TODOs, I'll be implementing some here, some to htm.core

nab/detectors/htmcore/README.md

breznak · 2019-07-24T15:32:32Z

nab/detectors/htmcore/htmcore_detector.py

+
+  def getAdditionalHeaders(self):
+    """Returns a list of strings."""
+    return ["raw_score"] #TODO add "prediction"


TODO output also predictions

nab/detectors/htmcore/htmcore_detector.py

breznak · 2019-07-24T15:39:50Z

nab/detectors/htmcore/htmcore_detector.py

+    rawScore = 0.5 #FIXME result.inferences["anomalyScore"]
+
+    # Update min/max values and check if there is a spatial anomaly
+    spatialAnomaly = False #TODO make this computed in SP (and later improve)


I like the idea of SpatialAnomaly being moved to SP itself SP return const random representation on empty input & SP anomaly htm.core#550

I don't 100% like the current implementation of spatial anomalies, but that can be improved later.

done in htm-community/htm.core#587

breznak · 2019-07-24T15:41:45Z

nab/detectors/htmcore/htmcore_detector.py

+      # Compute log(anomaly likelihood)
+      anomalyScore = self.anomalyLikelihood.anomalyProbability(inputData["value"], rawScore, inputData["timestamp"])
+      logScore = self.anomalyLikelihood.computeLogLikelihood(anomalyScore)
+      finalScore = logScore #TODO returls logScore and not probability? Fix that in our Likelihood.cpp; #TODO TM to provide anomaly {none, raw, likelihood} 


we should probably return logScore in Likelihood by default review and refactor AnomalyLikelihood code (both c++, python) htm.core#469

provide enum modes {none, raw, likelihood} to TM.anomaly

both done in htm-community/htm.core#588

nab/detectors/htmcore/htmcore_detector.py

breznak · 2019-07-24T15:46:21Z

nab/detectors/htmcore/setup.py

+
+    curr_dir = os.getcwd()
+    os.chdir(REPO_DIR)
+    os.system("git clone --depth=5 https://github.com/htm-community/htm.core")


when htm.core can be available on PyPI, skip all this mess htm-community/htm.core#19

breznak · 2019-07-24T15:46:46Z

run.py

                    nargs="*",
                    type=str,
-                    default=["numenta", "numentaTM", "htmjava", "null", "random",
+                    default=["numenta", "numentaTM", "htmcore", "htmjava", "null", "random",


new htmcore detector!

params and initialization same as in hotgym.py example in htm.core repo. With one expection: RDSE resulotion changed to match one of numenta_detector, not the former value swarmed in hotgym.

implemented the model and runs nice! (Model same as hotgym.py)

as unused in this (cleaner) HTM model.

not so good, need to optimize parameters.

breznak

The new NAB detector with our community HTM implementation (htmcore) is ready.

@dkeeney @ctrl-z-9000-times please review, everything is working, additional task would be improving params for our model

breznak · 2019-07-25T11:47:49Z

README.md

+| [KNN CAD](https://github.com/htm-community/NAB/tree/master/nab/detectors/knncad)&dagger; | 58.0     | 43.4  | 64.8     | | |
+| [Relative Entropy](http://www.hpl.hp.com/techreports/2011/HPL-2011-8.pdf) | 54.6 | 47.6 | 58.8 | | |
+| [Random Cut Forest](http://proceedings.mlr.press/v48/guha16.pdf) **** | 51.7 | 38.4 | 59.7 | | |
+| [htm.core](https://github.com/htm-community/htm.core/)	| 50.83 | 49.95 | 52.64 | `htmcore` |  |


first shot results are not really good, we have to pump it up guys! 📌

I've pushed commit that uses numenta_detector's params, currently running benchmarks

breznak · 2019-07-25T11:48:46Z

nab/detectors/htmcore/htmcore_detector.py

+SPATIAL_TOLERANCE = 0.05
+
+## parameters to initialize our HTM model (encoder, SP, TM, AnomalyLikelihood)
+# TODO optional: optimize these params, either manually and/or swarming. But first keep comparable to numenta_detector


TODO optimize params taken from hotgym

The parameters for the original NAB are written down, somewhere in the original nupic.

parameters for the original NAB

yes, I've created params from the orig nab numenta_detector. But we do not score as well (localAreaDensity instead of removed numActiveColsPerInhArea, more importantly removed BacktrackingTM, which had best results on NAB,...)

Ok, ive started looking at and playing with these parameters, and wow it runs fast! I am optimizing a few of the parameters by hand and will push the results soon. Then i can see about hooking this up to the optimization framework

I am optimizing a few of the parameters by hand and will push the results soon

please do. I've been trying to do some hand-optimizations, but I was still getting much worse results than Numenta's HTM :/

Here is the new current score. It's just a few points less than the target.

"reward_low_FN_rate": 66.1922106936328,

"reward_low_FP_rate": 58.7930712907694,

"standard": 63.081419488725054

new current score. It's just a few points less than the target.

that does look good! 👍 Let's merge this so we can go from there..

nab/detectors/htmcore/htmcore_detector.py

for information only, not used

use the same params as much as possible to have comparison, print statistics

FIXME for anomaly computation is not needed, as TM provides TM.anomaly already!

breznak added 3 commits July 24, 2019 14:54

HTM.core new detector installed

12d98e5

install htm.core detector from setup.py if installed, just skip, otherwise build from github

Merge branch 'py2_compatibility' into htm_community_detector

2d692db

Merge branch 'py2_compatibility' into htm_community_detector

e4e12c2

breznak added the enhancement New feature or request label Jul 24, 2019

breznak self-assigned this Jul 24, 2019

breznak added 2 commits July 24, 2019 16:26

Merge branch 'master' into htm_community_detector

b08c92e

new HtmcoreDetector POC

8de00c6

runable POC for community HTM detector from famous htm.core repo!

breznak commented Jul 24, 2019

View reviewed changes

Htmcore: updated Readme for this detector

8e62c87

breznak mentioned this pull request Jul 24, 2019

TM anomaly: new modes: likelihood,... htm-community/htm.core#588

Merged

3 tasks

breznak added 6 commits July 25, 2019 09:57

Htm.core detector: cleanup handleRecord()

1649ec4

Htm.core detector: params and Enc, SP, TM initialized

ae84be0

params and initialization same as in hotgym.py example in htm.core repo. With one expection: RDSE resulotion changed to match one of numenta_detector, not the former value swarmed in hotgym.

Htm.core runable detector!

7b49b09

implemented the model and runs nice! (Model same as hotgym.py)

Htm.core detector remove relic from numenta's Model API

02bbf1f

as unused in this (cleaner) HTM model.

Updated README

3c11d61

README: add htmcore detector's results

a1c262f

not so good, need to optimize parameters.

breznak changed the title ~~HTM community detector (py3) WIP~~ HTM community detector (py3) Jul 25, 2019

breznak added the model detector, model label Jul 25, 2019

breznak commented Jul 25, 2019

View reviewed changes

breznak and others added 4 commits July 25, 2019 14:05

Htmcore detector: add model params from numenta_detector

c893cb2

for information only, not used

Htmcore: add params comparable with numenta detector

4f2e8a5

use the same params as much as possible to have comparison, print statistics

remove wrong comment

01f4c41

FIXME for anomaly computation is not needed, as TM provides TM.anomaly already!

Improved parameters for htm.core detector.

3097a72

ctrl-z-9000-times approved these changes Jul 29, 2019

View reviewed changes

ctrl-z-9000-times merged commit b2f7862 into master Jul 29, 2019

ctrl-z-9000-times deleted the htm_community_detector branch July 29, 2019 13:49

Conversation

breznak commented Jul 24, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

breznak left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

breznak left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

breznak commented Jul 24, 2019 •

edited

Loading