Pages: [1] 2 3 ... 5
Author Topic: New open source tuning software  (Read 9097 times)
willemml
Full Member
***

Karma: +10/-0
Offline Offline

Posts: 50


« on: February 12, 2025, 05:09:17 PM »

Hey there!

While I wait for a spare ECU and some bench flashing hardware to arrive so I can start actually tuning, I played around with TunerPro in a VM a bit.

Anyways, I didn't like that I had to use windows to do the tuning and view diffs between tunes while reading the community projects (trying to learn what is being changed and why).
So I decided to write my own tuning software that works on all platforms. Currently all it does is load a bin, an XDF, and then display all the maps. It does have the ability to write bins, but there isn't a button for that (yet).
Currently it runs purely in the terminal, which is nice for me as it's a purely keyboard based UI.

All in all, its a very very WIP piece of software, and as such I am not distributing any executables, but it is there and in active development, I plan on adding built in checksumming, and possibly built in flashing (hopefully both bootmode and kwp2000). The goal is to make a very simple open source (and more importantly, multi platform) package for tuning Bosch's ME7 ECUs.

For now I have called it "HEXTuner" (I happened to have a hex editor opened) and the source is available on GitHub here
Logged
willemml
Full Member
***

Karma: +10/-0
Offline Offline

Posts: 50


« Reply #1 on: February 18, 2025, 07:56:58 AM »

It turns out that displaying large tables in a terminal window gets a bit confusing and unruly. So I have switched to a graphical interface. Editing doesn't work yet but tables can be displayed.
Logged
willemml
Full Member
***

Karma: +10/-0
Offline Offline

Posts: 50


« Reply #2 on: February 19, 2025, 11:25:10 AM »

There is now a fully functional GUI, it isn't pretty but it allows tiling editing windows (tables/scalars) and editing/writing to bins now works.
Logged
willemml
Full Member
***

Karma: +10/-0
Offline Offline

Posts: 50


« Reply #3 on: February 23, 2025, 11:30:58 PM »

Added some rudimentary support for getting categories from XDF files and displaying them. Does not work well with nested categories (un-nests them).

I am probably going to create my own file format and have a built in converter for XDFs. This will eliminate most of the road blocks I've been running into. Unfortunately the XDF format seems to be very hacky (though I dislike XML/heavily generic data structures given my primarily embedded background, so I am biased) and a lot of things seem like they were an afterthought.

Also I will try to remove axis definitions, I already support editing axis directly from an opened table, so having the extra tables makes little sense.

Also added in file pickers, so the bin/xdf paths aren't hard coded.
« Last Edit: February 23, 2025, 11:35:11 PM by willemml » Logged
_nameless
Hero Member
*****

Karma: +347/-802
Offline Offline

Posts: 2859



« Reply #4 on: February 24, 2025, 12:42:23 AM »

Cool, project. Have not had the chance to look into it but by the sounds of it you are keeping up on the project, nice job.
Logged

If you are broke or expecting free handouts DO NOT message me. I'll probably put you on blast if you do.
willemml
Full Member
***

Karma: +10/-0
Offline Offline

Posts: 50


« Reply #5 on: February 24, 2025, 06:54:59 PM »

Thanks. It is a fun project, and I get to look forward to eventually not needing to use my windows laptop at all. KWP flashing will be interesting when I get around to it. Gives me something easy to do in between tweaking my tune and trying to learn how ME7 works. Once I have a better understanding of tuning I'll probably move on to writing my own assembly patches, which should be much easier as I already have a strong understanding of low level computing (having written several instruction sets/architectures of my own). It would be nice to give myself some more room in some of the maps just as a starter project.

Hopefully other people will find it useful when its more mature, although I do not recommend using it right now, while everything does work, it's very clunky and lacks some important QoL features (like mass editing cells, comparing bins, graphing...)

Next up I will be adding graphs though.
« Last Edit: February 24, 2025, 07:29:43 PM by willemml » Logged
nyet
Administrator
Hero Member
*****

Karma: +609/-169
Online Online

Posts: 12290


WWW
« Reply #6 on: February 24, 2025, 09:36:14 PM »

Love what you are trying to do here.... a standalone linux based kwp flasher would be amazing... or an open source version of me7logger than can run on say, a pi
Logged

ME7.1 tuning guide
ECUx Plot
ME7Sum checksum
Trim heatmap tool

Please do not ask me for tunes. I'm here to help people make their own.

Do not PM me technical questions! Please, ask all questions on the forums! Doing so will ensure the next person with the same issue gets the opportunity to learn from your ex
willemml
Full Member
***

Karma: +10/-0
Offline Offline

Posts: 50


« Reply #7 on: February 25, 2025, 12:20:07 AM »

That is the end goal, most development is happening from an arm64 MacBook running Linux. I eventually want to get kwp flashing and logging from something embedded like an ESP32. Once I figure out how kwp works over serial with these vag kkl cables I'm sure it will be fairly straightforward.

My next step is to assemble the bench harness I got parts for and then flash the stock software to my unitronic ECU so I don't have to worry about any oddities with that. I don't have much use for the unitronic software anymore anyways, my own tune seems to run noticeably (hesitant to say much) smoother even with my very minimal knowledge from a couple weeks reading this forum and the wiki. Glad the PO payed for the OTS tune and not me.
Once I have the bench setup I'll be able to iterate pretty quickly with flashing/logging implementations.

Also, I added graphing capabilities earlier this evening before I went and messed with the load tables in my car. I haven't actually pushed the code yet though. I will probably do that tomorrow after some cleanup.

A lot of this wouldn't be possible in such short amounts of time without the amazing amount of libraries people have made for rust. A lot of things (like the graphing) are turned into an exercise of reading documentation instead of designing and implementing entire packages. This is just to say that I can't take credit for everything.
« Last Edit: February 25, 2025, 12:21:39 AM by willemml » Logged
willemml
Full Member
***

Karma: +10/-0
Offline Offline

Posts: 50


« Reply #8 on: February 25, 2025, 06:46:40 PM »

Well I was planning on testing some KWP2000 stuff tonight, that might have to wait. I put together the bench harness, but wired everything upside down! That's a bit embarrassing... Oh well. Might have killed the ECU, will put it in the car when I get a chance to see if I actually killed it or if I just have the harness wired wrong still. Hoping it's the latter so I don't "have" to buy a third. I could do all of this in the car, but it would be much nicer to be able to do the development from my desk.
« Last Edit: February 25, 2025, 06:51:50 PM by willemml » Logged
willemml
Full Member
***

Karma: +10/-0
Offline Offline

Posts: 50


« Reply #9 on: February 27, 2025, 11:05:03 AM »

ECU is fine, something wrong with the harness. Probably missed something in all the guides I read...

Question 1: so while I am heavily leaning towards designing my own open file format for map definitions, does anybody have recommendations of existing open formats to consider? Even if I don't use it directly I may write a converter so this app can use it.

Question 2: how does ME7Info actually work? If I write a logger I would want to implement its functionality.

Also for now I am avoiding reading any GPL code, as my goal is to release this as MIT or BSD0, but that may change if/when I make a final decision on licensing and well I progress without looking at the existing projects like the Nefmoto flasher. Ultimately I don't really care what the license is, but it would be nice if it were truly free.
Logged
nyet
Administrator
Hero Member
*****

Karma: +609/-169
Online Online

Posts: 12290


WWW
« Reply #10 on: February 27, 2025, 11:34:21 AM »

Question 1: so while I am heavily leaning towards designing my own open file format for map definitions, does anybody have recommendations of existing open formats to consider? Even if I don't use it directly I may write a converter so this app can use it.

XDF is the defacto "open" standard but it is cumbersome and has issues. ecuxplot has a kp->xdf converter you can look at as a reference if you like.

KP is totally useless. Binary blobs, proprietary, closed, opaque. Total garbage. I detest companies like EVC who show zero interest in anything but their own bottom line.

Quote
Question 2: how does ME7Info actually work? If I write a logger I would want to implement its functionality.

Combination of disassembly and pattern matching. prj has written a much much better tool... dont have the link handy

Quote
Also for now I am avoiding reading any GPL code, as my goal is to release this as MIT or BSD0, but that may change if/when I make a final decision on licensing and well I progress without looking at the existing projects like the Nefmoto flasher. Ultimately I don't really care what the license is, but it would be nice if it were truly free.

meh, im the defacto "owner" of the flasher. I am considering a license shift since the author has mostly abdicated it to me. Let me know if you feel like there cleanroom implementation problems (from a license standpoint). I'm sure we can work something out trivially.
Logged

ME7.1 tuning guide
ECUx Plot
ME7Sum checksum
Trim heatmap tool

Please do not ask me for tunes. I'm here to help people make their own.

Do not PM me technical questions! Please, ask all questions on the forums! Doing so will ensure the next person with the same issue gets the opportunity to learn from your ex
prj
Hero Member
*****

Karma: +1090/-512
Offline Offline

Posts: 6159


« Reply #11 on: February 27, 2025, 01:09:21 PM »

Combination of disassembly and pattern matching. prj has written a much much better tool... dont have the link handy

ME7Info works by parsing the structure defined in TKMWL.
Tool I wrote for pattern matching was an idea, but it's quite useless tbh, as you needed to define patterns in xml.

Internally these days I have something much better.
It can take any C166/ST10/PowerPC/TriCore binary, automatically detect the code areas and registers.
It can then disassemble it in-process, and with a bunch of clever algos and false positive rejection go through the code on a known and unknown binary and match all the memory accesses.
It tracks subroutines, calls, jumps, registers, copies between registers and so on.

This is the basis of being able to provide the Logger advanced service, because it is impossible to have the exact A2L for every file.
Been working on it for years. Unfortunately I don't think I can ever opensource it nor does it find maps, because I never intended for it to do that (I just work with measurements).

As shown in the screenshot, this takes just 4 seconds on ME7 on my mid tier laptop.

Alternative when you have a lot more time, is to use bindiff in Ghidra or Ida. It can also do a lot for finding symbols between files.
This of course takes much longer, as you have to load every binary correctly and process it. So what takes my tool under 30 seconds can easily take an hour using this method.
Good if you need to get the addresses for one car, but not viable commercially.
Logged

PM's will not be answered, so don't even try.
Log your car properly - WinOLS database - Tools/patches
willemml
Full Member
***

Karma: +10/-0
Offline Offline

Posts: 50


« Reply #12 on: February 27, 2025, 04:00:51 PM »

I've always wanted to write a disassembler. At least I won't be short of projects for a while. I really just need to write the KWP flasher and logger. I will just use ecu files from ME7Info. For now at least.

prj: it sounds like you wrote an automated C166 emulator/debugger combo.

I wonder if I could do some analysis knowing what input/output pins do what and then I would know which subroutines read/write the corresponding registers. Then using the FR to find out what functions use what I could probably map each function to a group of linked subroutines that access the required info... Much easier said than done of course, and I am dumbing it down from the full idea. Finding execution patterns would probably be easier than finding relevant IO access. Anyways, both require writing what would effectively be a disassembler and emulator.

Good to know about the formats. I will just create a documented format of my own. Probably using TOML. XDF has been hard to find documentation for.

nyet: I'll let you know how I fare in my "cleanroom" flasher implementation.
« Last Edit: February 27, 2025, 04:11:24 PM by willemml » Logged
nyet
Administrator
Hero Member
*****

Karma: +609/-169
Online Online

Posts: 12290


WWW
« Reply #13 on: February 27, 2025, 04:14:37 PM »

nyet: I'll let you know how I fare in my "cleanroom" flasher implementation.

I'm saying you don't have to Smiley nobody is going to sue you
Logged

ME7.1 tuning guide
ECUx Plot
ME7Sum checksum
Trim heatmap tool

Please do not ask me for tunes. I'm here to help people make their own.

Do not PM me technical questions! Please, ask all questions on the forums! Doing so will ensure the next person with the same issue gets the opportunity to learn from your ex
willemml
Full Member
***

Karma: +10/-0
Offline Offline

Posts: 50


« Reply #14 on: February 27, 2025, 04:19:09 PM »

I'm saying you don't have to Smiley nobody is going to sue you

Ahhh I misunderstood. Thanks Smiley
Logged
Pages: [1] 2 3 ... 5
  Print  
 
Jump to:  

Powered by SMF 1.1.21 | SMF © 2015, Simple Machines Page created in 0.023 seconds with 16 queries. (Pretty URLs adds 0s, 0q)