NefMoto > Technical > Data Logging > trim heatmap tool for tuning KFKHFM
Pages: [1]
« previous next »
Author Topic: trim heatmap tool for tuning KFKHFM  (Read 11625 times)
nyet
Administrator
Hero Member
*****

Karma: +609/-169
Online Online

Posts: 12301


WWW
« on: October 24, 2024, 01:20:02 AM »
Similar to the FKVVS tuner, this is specifically made to tune KFKHFM based on fr/frm logs.

https://github.com/nyetwurk/trim-heatmap/releases

It is command line based, so you have to give it one or more log files on the command line. If you do not know what that means, this tool is not for you



Code:
usage: heatmap.py [-h] [-w WINDOW] [-l LOAD_FILTER] [-r RPM_FILTER] [-m MIN_SAMPLES] [-f] [-u] [-v] [filename ...]

Create trim heatmap for KFKHFM based on fr/frm datalog

positional arguments:
  filename              csv files(s) to parse (log.csv)

options:
  -h, --help            show this help message and exit
  -w WINDOW, --window WINDOW
                        number of sequential rows to detect constant rpm/load (5)
  -l LOAD_FILTER, --load-filter LOAD_FILTER
                        change in load which is still "constant" load (10)
  -r RPM_FILTER, --rpm-filter RPM_FILTER
                        change in RPM which is still "constant" RPM (100)
  -m MIN_SAMPLES, --min-samples MIN_SAMPLES
                        minimum number of samples required to generate a cell (10)
  -f, --use-fr          use "fr" instead of "frm" (the default)
  -u, --use-unweighted-mean
                        use unweighted mean instead of weighted average (the default)
  -v, --verbose
« Last Edit: October 24, 2024, 01:22:52 AM by nyet » Logged
ME7.1 tuning guide
ECUx Plot
ME7Sum checksum
Trim heatmap tool

Please do not ask me for tunes. I'm here to help people make their own.

Do not PM me technical questions! Please, ask all questions on the forums! Doing so will ensure the next person with the same issue gets the opportunity to learn from your ex
ratosluaf
Jr. Member
**

Karma: +1/-3
Offline Offline

Posts: 39
« Reply #1 on: January 06, 2025, 03:31:43 AM »
I've done something similar in the past for myself.
Now I had some free time so I've rewritten it to streamlit and deployed it on the community cloud.

Here's the link: https://fixhfm.streamlit.app/

It does depend on nmot_w, rl_w and fr_w from me7logger, so idk if it will work with all logs. It does stick your data to nearest nmot_w and rl_w in kfkhfm table and calculate it depending on weighted mean.

You can copy your table from tunerpro or import your table from csv file - example of log file and map csv file in github repo.

github repo: https://github.com/rwalkuski/hfm_fixer

Any feedback is welcome since now i have no possibility to test output myself, but had success with my own car in the past.


Edit: for now, when you want to copy from site to tunerpro, please buffer the data in notepad or any other text editor Copying directly into tunerpro causes destroying of data table, but i don't know why for now.
edit2: copying directly from site lacks of carriage return which tunerpro interprets as new line. At the moment, i do not know how to fix that, so please buffer your data.
« Last Edit: January 06, 2025, 05:09:25 AM by ratosluaf » Logged
Crazy18T
Newbie
*

Karma: +0/-0
Offline Offline

Posts: 14
« Reply #2 on: April 29, 2025, 11:20:31 PM »
Finally have my car back together (in a slightly temporary fashion), so I was eager to try this. Smiley After struggling to get the output to not cause issues with the tune (tried every setting and many iterations), I decided to take a closer look at the code. It seems the bucket creation code is quite broken.

Applying this patch, we can see both RPM and load bucket calculation are way off:
Code:
diff -ur trim-heatmap-0.0.5/src/heatmap.py trim-heatmap-0.0.5-testing/src/heatmap.py
--- trim-heatmap-0.0.5/src/heatmap.py	2024-11-06 02:39:02.000000000 -0700
+++ trim-heatmap-0.0.5-testing/src/heatmap.py	2025-04-29 23:49:57.097326680 -0600
@@ -1,4 +1,4 @@
-#!/usr/bin/env python
+#!/usr/bin/env python3
 
 # internal modules
 import os
@@ -78,6 +78,7 @@
 		frdata[lk]={}
 		for rk,rv in rpms.items():
 			query = f"{rv[0]} <= @lc.rpm <= {rv[1]} & {lv[0]} <= @lc.load <= {lv[1]}"
+			print(query)
 			res = lc.query(query).copy()
 			if len(res.index) >= (args.min_samples, 1)[args.no_filter]: # ternary is (false, true)
 				# size of cell, center to corner

A few rounds of output with patch applied (example log.csv):
Quote
$ ./heatmap.py
0 <= @lc.rpm <= 859.0 & 0 <= @lc.load <= 14.0
860.0 <= @lc.rpm <= 1119.0 & 0 <= @lc.load <= 14.0
1120.0 <= @lc.rpm <= 1379.0 & 0 <= @lc.load <= 14.0
1380.0 <= @lc.rpm <= 1759.0 & 0 <= @lc.load <= 14.0
1760.0 <= @lc.rpm <= 2259.0 & 0 <= @lc.load <= 14.0
2260.0 <= @lc.rpm <= 2759.0 & 0 <= @lc.load <= 14.0
2760.0 <= @lc.rpm <= 3259.0 & 0 <= @lc.load <= 14.0
3260.0 <= @lc.rpm <= 3759.0 & 0 <= @lc.load <= 14.0
3760.0 <= @lc.rpm <= 4259.0 & 0 <= @lc.load <= 14.0
4260.0 <= @lc.rpm <= 4759.0 & 0 <= @lc.load <= 14.0
4760.0 <= @lc.rpm <= 5259.0 & 0 <= @lc.load <= 14.0
5260.0 <= @lc.rpm <= 5759.0 & 0 <= @lc.load <= 14.0
5760.0 <= @lc.rpm <= 6259.0 & 0 <= @lc.load <= 14.0
6260.0 <= @lc.rpm <= 10000 & 0 <= @lc.load <= 14.0
0 <= @lc.rpm <= 859.0 & 15.0 <= @lc.load <= 24.125
860.0 <= @lc.rpm <= 1119.0 & 15.0 <= @lc.load <= 24.125
1120.0 <= @lc.rpm <= 1379.0 & 15.0 <= @lc.load <= 24.125
1380.0 <= @lc.rpm <= 1759.0 & 15.0 <= @lc.load <= 24.125
1760.0 <= @lc.rpm <= 2259.0 & 15.0 <= @lc.load <= 24.125
2260.0 <= @lc.rpm <= 2759.0 & 15.0 <= @lc.load <= 24.125
2760.0 <= @lc.rpm <= 3259.0 & 15.0 <= @lc.load <= 24.125
3260.0 <= @lc.rpm <= 3759.0 & 15.0 <= @lc.load <= 24.125
3760.0 <= @lc.rpm <= 4259.0 & 15.0 <= @lc.load <= 24.125
4260.0 <= @lc.rpm <= 4759.0 & 15.0 <= @lc.load <= 24.125
4760.0 <= @lc.rpm <= 5259.0 & 15.0 <= @lc.load <= 24.125
5260.0 <= @lc.rpm <= 5759.0 & 15.0 <= @lc.load <= 24.125
5760.0 <= @lc.rpm <= 6259.0 & 15.0 <= @lc.load <= 24.125
6260.0 <= @lc.rpm <= 10000 & 15.0 <= @lc.load <= 24.125
0 <= @lc.rpm <= 859.0 & 25.125 <= @lc.load <= 33.875
860.0 <= @lc.rpm <= 1119.0 & 25.125 <= @lc.load <= 33.875
1120.0 <= @lc.rpm <= 1379.0 & 25.125 <= @lc.load <= 33.875
1380.0 <= @lc.rpm <= 1759.0 & 25.125 <= @lc.load <= 33.875
1760.0 <= @lc.rpm <= 2259.0 & 25.125 <= @lc.load <= 33.875
2260.0 <= @lc.rpm <= 2759.0 & 25.125 <= @lc.load <= 33.875
2760.0 <= @lc.rpm <= 3259.0 & 25.125 <= @lc.load <= 33.875
3260.0 <= @lc.rpm <= 3759.0 & 25.125 <= @lc.load <= 33.875
3760.0 <= @lc.rpm <= 4259.0 & 25.125 <= @lc.load <= 33.875
4260.0 <= @lc.rpm <= 4759.0 & 25.125 <= @lc.load <= 33.875
4760.0 <= @lc.rpm <= 5259.0 & 25.125 <= @lc.load <= 33.875
5260.0 <= @lc.rpm <= 5759.0 & 25.125 <= @lc.load <= 33.875
5760.0 <= @lc.rpm <= 6259.0 & 25.125 <= @lc.load <= 33.875
6260.0 <= @lc.rpm <= 10000 & 25.125 <= @lc.load <= 33.875
0 <= @lc.rpm <= 859.0 & 34.875 <= @lc.load <= 44.0
860.0 <= @lc.rpm <= 1119.0 & 34.875 <= @lc.load <= 44.0
1120.0 <= @lc.rpm <= 1379.0 & 34.875 <= @lc.load <= 44.0
1380.0 <= @lc.rpm <= 1759.0 & 34.875 <= @lc.load <= 44.0
1760.0 <= @lc.rpm <= 2259.0 & 34.875 <= @lc.load <= 44.0
2260.0 <= @lc.rpm <= 2759.0 & 34.875 <= @lc.load <= 44.0
2760.0 <= @lc.rpm <= 3259.0 & 34.875 <= @lc.load <= 44.0
3260.0 <= @lc.rpm <= 3759.0 & 34.875 <= @lc.load <= 44.0
3760.0 <= @lc.rpm <= 4259.0 & 34.875 <= @lc.load <= 44.0
4260.0 <= @lc.rpm <= 4759.0 & 34.875 <= @lc.load <= 44.0
4760.0 <= @lc.rpm <= 5259.0 & 34.875 <= @lc.load <= 44.0
5260.0 <= @lc.rpm <= 5759.0 & 34.875 <= @lc.load <= 44.0
5760.0 <= @lc.rpm <= 6259.0 & 34.875 <= @lc.load <= 44.0
6260.0 <= @lc.rpm <= 10000 & 34.875 <= @lc.load <= 44.0
Logged
nyet
Administrator
Hero Member
*****

Karma: +609/-169
Online Online

Posts: 12301


WWW
« Reply #3 on: April 29, 2025, 11:57:17 PM »
post the log ill take a look

what are you expecting to see?
« Last Edit: April 29, 2025, 11:59:11 PM by nyet » Logged
ME7.1 tuning guide
ECUx Plot
ME7Sum checksum
Trim heatmap tool

Please do not ask me for tunes. I'm here to help people make their own.

Do not PM me technical questions! Please, ask all questions on the forums! Doing so will ensure the next person with the same issue gets the opportunity to learn from your ex
Crazy18T
Newbie
*

Karma: +0/-0
Offline Offline

Posts: 14
« Reply #4 on: April 30, 2025, 12:03:42 AM »
This is the example log.csv as stated. I assume the bucket code is AI generated? Why do you not see the issue here?
Logged
Crazy18T
Newbie
*

Karma: +0/-0
Offline Offline

Posts: 14
« Reply #5 on: April 30, 2025, 12:34:06 AM »
I expect to see the RPM and load buckets populated as you provided (rpms loads) for the 2.7T. In my case they are adjusted for my setup, but that's beyond the scope of this thread. Sorry! I didn't mean to come across as rude.
Logged
nyet
Administrator
Hero Member
*****

Karma: +609/-169
Online Online

Posts: 12301


WWW
« Reply #6 on: April 30, 2025, 10:39:27 AM »
Quote from: Crazy18T on April 30, 2025, 12:34:06 AM
I expect to see the RPM and load buckets populated as you provided (rpms loads) for the 2.7T. In my case they are adjusted for my setup, but that's beyond the scope of this thread. Sorry! I didn't mean to come across as rude.

You're not printing the output. You're printing a query that happens in the core code, before anything else happens, including all processing. What is the input you are giving it? What command line parameters?
« Last Edit: April 30, 2025, 10:41:11 AM by nyet » Logged
ME7.1 tuning guide
ECUx Plot
ME7Sum checksum
Trim heatmap tool

Please do not ask me for tunes. I'm here to help people make their own.

Do not PM me technical questions! Please, ask all questions on the forums! Doing so will ensure the next person with the same issue gets the opportunity to learn from your ex
Crazy18T
Newbie
*

Karma: +0/-0
Offline Offline

Posts: 14
« Reply #7 on: April 30, 2025, 03:31:55 PM »
I'm aware I'm not printing the output, but the output can't be correct if the buckets/cells aren't populated using the proper values. The command line is included with my post. It's just showing what buckets are created using your example log.csv.
Logged
Crazy18T
Newbie
*

Karma: +0/-0
Offline Offline

Posts: 14
« Reply #8 on: April 30, 2025, 03:58:17 PM »
calc_ranges() seems to be completely broken. I don't understand how you aren't seeing this? It creates rpms as 14 buckets/cells as expected, but for example the first bucket shouldn't be 0-859. It should be 720-999. And it continues on with both bucket ranges in this confusing way.
Logged
nyet
Administrator
Hero Member
*****

Karma: +609/-169
Online Online

Posts: 12301


WWW
« Reply #9 on: April 30, 2025, 10:08:57 PM »
the map numbers are the *centers* of the cells. We either do a linear average or a weighted average by distance from the center of the cell (which is the map axis data)
« Last Edit: April 30, 2025, 10:11:22 PM by nyet » Logged
ME7.1 tuning guide
ECUx Plot
ME7Sum checksum
Trim heatmap tool

Please do not ask me for tunes. I'm here to help people make their own.

Do not PM me technical questions! Please, ask all questions on the forums! Doing so will ensure the next person with the same issue gets the opportunity to learn from your ex
Crazy18T
Newbie
*

Karma: +0/-0
Offline Offline

Posts: 14
« Reply #10 on: Yesterday at 10:04:27 PM »
Appreciate the clarification. As you probably figured out, my issues actually stemmed from minor hardware issues. I would like to continue on in helping to develop open source tools to help dial in air/fuel on ME7. I was able to create a KFLF version for ME7.5 1.8T no problem. The reason being, we have additional ranges on this load/air correction map that could help smooth things out.

The obvious next step is to create the inverse map using the same logic, so we can try and determine when we have an air/load correction situation vs just a non-linearity issue with injector pulse. My first attempt to do this works in continuous mode, but it provides no output in other modes. Do you think this is because calc_ranges() doesn't tolerate the small floating points used in IOT of FKKVS? It's the best guess I have at this point. Appreciate the code!

Quote
$ trim-heatmap-fkkvs/src/heatmap.py log.csv --text -s1
Empty DataFrame
Columns: []
Index: []
$ trim-heatmap-fkkvs/src/heatmap.py log.csv -s1
Traceback (most recent call last):
  File "/home/tuning/trim-heatmap-fkkvs/src/heatmap.py", line 255, in <module>
    main()
  File "/home/tuning/trim-heatmap-fkkvs/src/heatmap.py", line 246, in main
    sbs.heatmap(heatmap, annot=(not args.continuous), center=0, cmap='PiYG', cbar_kws={'label': '% trim'}).invert_yaxis()
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/seaborn/matrix.py", line 446, in heatmap
    plotter = _HeatMapper(data, vmin, vmax, cmap, center, robust, annot, fmt,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/seaborn/matrix.py", line 163, in __init__
    self._determine_cmap_params(plot_data, vmin, vmax,
  File "/usr/lib/python3/dist-packages/seaborn/matrix.py", line 202, in _determine_cmap_params
    vmin = np.nanmin(calc_data)
           ^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/numpy/lib/nanfunctions.py", line 343, in nanmin
    res = np.fmin.reduce(a, axis=axis, out=out, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: zero-size array to reduction operation fmin which has no identity
« Last Edit: Yesterday at 10:34:58 PM by Crazy18T » Logged
Pages: [1]
  Print  
« previous next »
 
Jump to:  

Navigation
Personal Tools
  • June 27, 2025, 07:45:22 PM
  • Welcome, Guest. Please login or register.
    Did you miss your activation email?

    Login with username, password and session length
Search
 
Advanced Search
About Disclaimer
All Rights Reserved 2010 Nefarious Motosports
Powered by SMF 1.1.21 | SMF © 2015, Simple Machines Page created in 0.685 seconds with 17 queries. (Pretty URLs adds 0.103s, 0q)