NefMoto

ECU Files => ECU Definition Files => Topic started by: prj on October 12, 2012, 10:05:52 AM



Title: ASM based map locator platform
Post by: prj on October 12, 2012, 10:05:52 AM
Time to give back to the forum.

(http://gm.mainframe.no/pics/maplocator.png)

In short - I have created an extensible tool, which allows you to locate maps in any binary based on the actual machine code.
It can then dump them to XDF or map pack format.
The idea is a bit similar to spen's tool (that he does not want to share), but the execution is most likely very different.

The tool supports 32 bit and 64 bit versions of Windows, Linux and Mac OS X.
On most platforms you can just double-click the maplocator-ui.jar file.
If you can't for some reason, start it with java -jar maplocator-ui.jar from the command line. This will also allow you to see debug output from the tool.

The UI is very simplistic, but underneath lies a full blown customizable detection engine.
Extensible both through plugins and XML's, the goal is to have people collaborate and make finding the most important maps on any ME7 binary as simple as pushing a button.
It is actually possible to support other ECU families as well, but for that plugins have to be written, as the XML plugin is geared towards a C167 architecture and ME7.

This is an early version. There is very little exception handling and map pack output is not supported yet.
There are a few XML definitions inside the me7xmls folder, that it uses to locate maps.

I will post a tutorial and documentation on how to create your own definitions over the weekend.
The main goal of this tool is to have a collaborative effort. There are quite a few people on the forum here, who could contribute XML definitions - I would never have enough time to do it all alone.
For now, feel free to play around.
The entire source of the application can be cloned from here:
https://github.com/prj/me7-tools (https://github.com/prj/me7-tools)

Run mvn package to build the application, which will be assembled and placed into maplocator-ui/maplocator-release.
You will need maven2+ and Java 6+ to build.

Changelog:
17.10.2012
- Support for multiple patterns
- Better axis length detection
- Added KFZW/KFZW2
- Maps that are found are now sorted alphabetically in the UI


Title: Re: ASM based map locator platform
Post by: prj on October 12, 2012, 10:08:38 AM
Part 1: Basic definitions

Definitions are stored in a XML format in the me7xmls directory.
There is also a file called "mapdef.xsd" which defines the schema according to which the XML files are created.
When creating definitions or editing existing ones, I strongly recommend you use an editor which supports schema and auto completing, as it will make your life a lot easier.

Let's take a look at locating the 16 bit KRKTE.
The first thing, is to find the KRKTE access in IDA Pro. This is simple - pick a binary you have a DAMOS for, and then go from there.
I will use the 551K RS4 binary for this example.

In this binary KRKTE is at 0x1C9DC. There are three places which access this value in the binary.
So we must choose one of them and concentrate on it. It is best to look for a spot that has unique code around it.
Like this one:
(http://gm.mainframe.no/pics/krkte1.png)

Time to create the initial XML:

<?xml version="1.0" encoding="UTF-8"?>
<map xmlns="http://prj-tuning.com/mapdef" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://prj-tuning.com/mapdef mapdef.xsd">
</map>


The first thing to add is the "ID". The "ID" is a unique identifier, which identifies the map.
There can be multiple XML files/definitions for the same ID. It is a good idea to name the file the same as the ID, so in this case KRKTE.xml.

Let's add the ID:
<?xml version="1.0" encoding="UTF-8"?>
<map xmlns="http://prj-tuning.com/mapdef" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://prj-tuning.com/mapdef mapdef.xsd">
   <id>KRKTE</id>
</map>


Now comes the most powerful feature of the map locator tool - the pattern.
The pattern is a way to locate a matching area in a binary. So we will have to create a pattern.

The pattern consists of the following building blocks:
HEX - bytes that should be matched exactly
XX - a byte that should be skipped.
XX<number> - means that zero to <number> bytes should be skipped.
Every time the pattern is matched, the offset that is returned is the address where the first character in the pattern matched.
If you would like to move the offset that is returned to an arbitrary place in the pattern, you can prefix any of the members of the pattern with MM.
The prefixed member will become the new reported offset.

For example F2 XX MMF2 XX.

Let's build a pattern for our location. It is a simple pattern, and comes out as follows:
F2 F4 XX XX 7C 44 E0 05 70 55.

We mask out the actual address from the pattern, because it will be different between binaries.
Let's add this to the xml:

<?xml version="1.0" encoding="UTF-8"?>
<map xmlns="http://prj-tuning.com/mapdef" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://prj-tuning.com/mapdef mapdef.xsd">
   <id>KRKTE</id>
   <pattern>F2 F4 XX XX 7C 44 E0 05 70 55</pattern>
</map>


Now the main building blocks have been added. It will find the pattern at an offset, however this is not terribly useful yet.
The way the C167 addresses memory is via a DPP and offset. So we have to specify where from the pattern location the DPP is located and where the offset is located.
In this case our DPP - 0x207h is located two bytes before the pattern start. If we omit this, the default DPP of 0x204h would be used.
Our offset - 0x09DC is located two bytes after the pattern start.

The total address is calculated as: dpp * 0x4000 - 0x800000 + offset. The subtraction is because of where the EEPROM is loaded.
Let's add this information to the XML:
<?xml version="1.0" encoding="UTF-8"?>
<map xmlns="http://prj-tuning.com/mapdef" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://prj-tuning.com/mapdef mapdef.xsd">
   <id>KRKTE</id>
   <pattern>F2 F4 XX XX 7C 44 E0 05 70 55</pattern>
   <address>
      <offset>2</offset>
      <dppOffset>-2</dppOffset>
   </address>
</map>


Alternatively a marker could be specified at F2 F4 MMXX XX... and the offset omitted, and DPP offset set to -4.

Now we just need to add the data to convert the located value to a legible form. This is done by the conversion element.

<?xml version="1.0" encoding="UTF-8"?>
<map xmlns="http://prj-tuning.com/mapdef" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://prj-tuning.com/mapdef mapdef.xsd">
   <id>KRKTE</id>
   <pattern>F2 F4 XX XX 7C 44 E0 05 70 55</pattern>
   <address>
      <offset>2</offset>
      <dppOffset>-2</dppOffset>
   </address>
   <conversion>
      <factor>0.000167</factor>
      <width>2</width>
      <endianness>LoHi</endianness>
   </conversion>
</map>


The factor is specified - the default factor is 1.0.
The offset is not needed - the default offset is 0.0, which suits us in this case.
The default width is 1 byte and the default endianness is HiLo. In this case the value is 2 bytes wide and bigendian, so we specify the endianness as LoHi.

And that's it - this XML is enough to detect KRKTE in almost any ME7 file, where it is a 16 bit value.
For 8 bit KRKTE's a slight modification is required, an example can be found in KRKTE_8.xml.


Title: Re: ASM based map locator platform
Post by: prj on October 12, 2012, 10:11:06 AM
Part 2: Advanced definitions

Part 1 dealt with the very basics and simple definitions.
Let's do some more advanced things.

The map I am going to pick for this example is KFLBTS.
There are a couple challenges to overcome as well.
1. Both axes of the map, and their lengths have to be located.
2. KFLBTS is different between binaries. In some binaries the load axis is 16 bit, in others it is 8 bit.

Let's first find the map access in the binary.
I will find this in two binaries. One will be ME7.1.1 on the RS6, and the other a 512k narrowband ME7.5 Audi TT.

RS6:
(http://gm.mainframe.no/pics/kflbts_rs6.png)

TT:
(http://gm.mainframe.no/pics/kflbts_tt.png)

There are a couple of issues with this.
1. The code is a bit different.
2. There is only one axis directly accessed, the other one is pre-loaded.

So let's try to write a pattern:
88 XX E6 FC MMXX XX E6 FD XX XX E6 FE XX XX E6 FF XX XX DA XX XX XX 08 04 XX2 5C 54 D7

Note the MM - this is the marker. A lot of things are masked out.
What uniquely identifies this area are the following instructions:
add r0, #4
shl r4, #5

This is all good and fine on the TT ECU, but the RS6 ECU has some data in the middle.
Luckily there is the "XX<number>" keyword, which allows to ignore 0 to <number> bytes. So this is used to skip the bytes on a RS6 ecu, and force a match.

Let's define the main map location:
<map xmlns="http://prj-tuning.com/mapdef" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://prj-tuning.com/mapdef mapdef.xsd ">
   <id>KFLBTS</id>
   <pattern>88 XX E6 FC MMXX XX E6 FD XX XX E6 FE XX XX E6 FF XX XX DA XX XX XX 08 04 XX2 5C 54 D7</pattern>
   <address>
      <dppOffset>4</dppOffset>
   </address>
   <conversion>
      <factor>0.007813</factor>
   </conversion>
</map>


The address offset is not needed, because our MM marker is right at where the address is.
The only things that have to be specified are the DPP offset, which is 4, and the conversion factor.
Everything else matches defaults (1 byte wide, no offset, and so on).

Of course in this case this is a 3D map, so two more maps must be defined - the horizontal and the vertical axis.
The main map XML merely contains references to them:
<map xmlns="http://prj-tuning.com/mapdef" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://prj-tuning.com/mapdef mapdef.xsd ">
   <id>KFLBTS</id>
   <pattern>88 XX E6 FC MMXX XX E6 FD XX XX E6 FE XX XX E6 FF XX XX DA XX XX XX 08 04 XX2 5C 54 D7</pattern>
   <address>
      <dppOffset>4</dppOffset>
   </address>
   <conversion>
      <factor>0.007813</factor>
   </conversion>
   <rowAxis>
      <id>SRL12GKUW</id>
      <id>SRL12GKUB</id>
   </rowAxis>
   <colAxis>
      <id>SNM16GKUB</id>
   </colAxis>
</map>


Note two references for the row axis. This is because on some ECU's the row axis consists of words, on others it consists of bytes (2 and 1 wide).
Should both be found, then the first one is preferred. So this also acts as an order.

The next thing is to define the axis maps.
The RPM axis is easy. We can use the same pattern, and just specify a different offset.
Every pattern that is looked up on a binary is cached. So from a performance standpoint, it is better to re-use exactly the same pattern without any changes. Moving the marker would mean that the binary has to be searched through again for example.

We get the following definition:
<?xml version="1.0" encoding="UTF-8"?>
<map xmlns="http://prj-tuning.com/mapdef" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://prj-tuning.com/mapdef mapdef.xsd ">
   <id>SNM16GKUB</id>
   <pattern>88 XX E6 FC MMXX XX E6 FD XX XX E6 FE XX XX E6 FF XX XX DA XX XX XX 08 04 XX2 5C 54 D7</pattern>
   <address>
      <offset>8</offset>
      <dppOffset>12</dppOffset>
   </address>
   <conversion>
      <factor>40</factor>
   </conversion>
</map>


With the second axis there is a bit more trouble. We must find it separately using a different pattern.
And the pattern will be different for both the 16 bit and the 8 bit axis.

16 bit:
<?xml version="1.0" encoding="UTF-8"?>
<map xmlns="http://prj-tuning.com/mapdef" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://prj-tuning.com/mapdef mapdef.xsd ">
   <id>SRL12GKUW</id>
   <pattern>F6 8E XX XX E6 FC XX XX E6 FD XX XX C2 FE XX50 E6 FC MMXX</pattern>
   <address>
      <dppOffset>4</dppOffset>
   </address>
   <conversion>
      <width>2</width>
      <endianness>LoHi</endianness>
      <factor>0.023438</factor>
   </conversion>
</map>


8 bit:
<?xml version="1.0" encoding="UTF-8"?>
<map xmlns="http://prj-tuning.com/mapdef" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://prj-tuning.com/mapdef mapdef.xsd ">
   <id>SRL12GKUB</id>
   <pattern>E6 FC XX XX E6 FD XX XX C2 FE XX XX D7 40 XX XX F2 FF XX XX DA XX XX XX D7 40 XX XX F6 F4 XX XX DB 00</pattern>
   <address>
      <offset>2</offset>
      <dppOffset>6</dppOffset>
   </address>
   <conversion>
      <factor>0.75</factor>
   </conversion>
</map>


If a map is assigned as an axis to another map, and it does not contain a length element, the length is automatically assumed to be the first value at the address, and this is processed appropriately.
However, this is not useful for all maps. For example for LAMFA the lengths are stored in a different place, which is pointed by code.

In this case the length element helps.
<?xml version="1.0" encoding="UTF-8"?>
<map xmlns="http://prj-tuning.com/mapdef" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://prj-tuning.com/mapdef mapdef.xsd ">
   <id>LAMFA_COLAXIS</id>
   <pattern>F2 F4 XX XX 88 40 C2 F5 XX XX 88 50 E6 F4 MMXX XX E6 F5</pattern>
   <address>
      <offset>12</offset>
      <dppOffset>16</dppOffset>
   </address>
   <conversion>
      <factor>0.003052</factor>
      <width>2</width>
      <endianness>LoHi</endianness>
   </conversion>
   <length>
      <address>
         <offset>44</offset>
         <dppOffset>40</dppOffset>
      </address>
   </length>
</map>


In this case there is a length element, which contains an address element.
The address element works exactly the same as the main address of the map.

This concludes the second part of the tutorial.


Title: Re: ASM based map locator platform
Post by: prj on October 12, 2012, 10:19:56 AM
Part 3: XML reference

To create a new XML definition, simply add an XML to the me7xmls folder.
You can keep the program running and re-parse a binary as often as you like. Changes to your XML's will be picked up on the fly.

The mapdef.xsd file outlays the format. Here is a list of all the elements and details on what they do.


Map:
<map> - the root map element.
   <id> - the map ID
   <pattern> - the lookup pattern to match the map. You can specify multiple pattern elements, they get checked from top to bottom.
   <afterRowAxis> - can be used instead of pattern, if the given map is located right after the row axis.
   <afterColAxis> - can be used instead of pattern, if the given map is located right after the column axis.
   <address> - address element, describes the map location
   <conversion> - conversion element, describes the map data.
   <length> - length element. Describes the length of the map if it needs to be loaded.
   <rowAxis> - Lists references to the row axis
   <colAxis> - Lists references to the column axis

Address:
<address> - the address element describes the address offset from the pattern location.
   <offset> - the offset from the matched pattern location/marker. Defaults to 0.
   <dpp> - hard coded dpp. Defaults to 0x204h, if dpp nor dppOfset is specified.
   <dppOffset> - the offset from the matched pattern for the dpp value.

Conversion:
<conversion> - the conversion element describes the map data.
   <factor> - the factor for map data, defaults to 1.0.
   <offset> - the offset for map data, defaults to 0.0.
   <width> - the map width. 1 or 2, defaults to 1.
   <endianness> - the map endianness. LoHi or HiLo, defaults to HiLo.
   <signed> - whether the value is a signed data type, defaults to false.
   <alt> - another Conversion element. This is used if an axis is specified as 16 bit, but is actually 8 bit. Look at SNM16ZUUB for an example.

Length:
<length> - the length element is used to specify the length of a map. For maps with axes this is calculated from axis lengths.
   <hardcoded> - hardcoded length inside the XML, defaults to 1.
   <address> - address element to locate the length in the binary.
   <width> - width element, only valid with address element, defines whether length is 1 or 2 bytes long.

RowAxis/ColAxis:
<rowAxis> - the rowAxis and colAxis elements specify a hierarchical list of axis candidate ID's.
   <id> - one or more ID's of the axis maps.



Part 4: Plugin reference

If you would to define a plugin, then you will have to implement the LocatorPlugin interface, make a jar, and put the jar into the "lib" folder.
The folder is scanned for plugins on startup.

The LocatorPlugin interface defines a single method:
public interface LocatorPlugin {
  Collection<? extends LocatedMap> locateMaps(byte[] binary);
}


LocatedMap is a bean, which your plugin should fill out as completely as possible.
Each plugin is run in a separate background thread.


Title: Re: ASM based map locator platform
Post by: professor on October 12, 2012, 11:00:24 AM
Music to my eyes  ;D


Title: Re: ASM based map locator platform
Post by: phila_dot on October 12, 2012, 11:05:10 AM
It is up to the community to create an extensive XML for this?



Title: Re: ASM based map locator platform
Post by: prj on October 12, 2012, 11:14:00 AM
It is up to the community to create an extensive XML for this?

Basically, yes.
I will obviously add more and more myself, but the entire thing is built with this in mind.
Every design decision was made with this in mind - the one-map-per-xml design, so people can easily post in little bits and so on.
My goal is/was to provide a stable platform rather than go through Motronic binaries in ASM ;)

Besides XML's it is also possible to write plugins very easily, if you want to detect things programmatically.
All you have to do is implement the LocatorPlugin interface in one of your classes, and drop the .jar into the lib folder.
It will pick this up on runtime and run your plugin. The XML based locator is just a plugin itself. An extensive one, but nonetheless just a plugin.


Title: Re: ASM based map locator platform
Post by: prj on October 12, 2012, 11:19:11 AM
It is up to the community to create an extensive XML for this?

P.S. You are one of the people I was targeting with this. Give me some feedback, what do you think? :)


Title: Re: ASM based map locator platform
Post by: prj on October 12, 2012, 01:00:42 PM
Added documentation.
It should be useful as a general guide on locating maps in ASM in ME7 as well.


Title: Re: ASM based map locator platform
Post by: nyet on October 12, 2012, 01:14:32 PM
awesome. git cloning now.


Title: Re: ASM based map locator platform
Post by: prj on October 12, 2012, 01:33:58 PM
awesome. git cloning now.

Use mvn package to build the entire thing....

------

Added CLRHK and CATR:
https://github.com/prj/me7-tools/tree/master/maplocator-ui/resources/me7xmls (https://github.com/prj/me7-tools/tree/master/maplocator-ui/resources/me7xmls)


Title: Re: ASM based map locator platform
Post by: prj on October 12, 2012, 01:37:27 PM
I have a few java plugins in mind as well.
Using the Pattern matcher a Java plugin should be written that will find and define all DTC's by code using the algorithm Tony posted a while back.

This will allow to disable DTC's very quickly.
Maybe nyet feels up to the task? :)


Title: Re: ASM based map locator platform
Post by: nyet on October 12, 2012, 01:39:19 PM
patch to fix some compile errors for me

Code:
diff --git a/maplocator-ui/pom.xml b/maplocator-ui/pom.xml
index 1fcbaca..c00c73c 100644
--- a/maplocator-ui/pom.xml
+++ b/maplocator-ui/pom.xml
@@ -32,6 +32,7 @@
  <plugin>
  <artifactId>maven-compiler-plugin</artifactId>
  <configuration>
+ <source>1.6</source>
  <target>1.6</target>
  </configuration>
  </plugin>
@@ -107,4 +108,4 @@
  </plugins>
  </build>
 
-</project>
\ No newline at end of file
+</project>
diff --git a/maplocator-xmlplugin/pom.xml b/maplocator-xmlplugin/pom.xml
index b3c0dbd..7d7fd0d 100644
--- a/maplocator-xmlplugin/pom.xml
+++ b/maplocator-xmlplugin/pom.xml
@@ -20,6 +20,13 @@
  <build>
  <plugins>
  <plugin>
+ <artifactId>maven-compiler-plugin</artifactId>
+ <configuration>
+ <source>1.6</source>
+ <target>1.6</target>
+ </configuration>
+ </plugin>
+ <plugin>
  <artifactId>maven-antrun-plugin</artifactId>
  <executions>
  <execution>
@@ -37,4 +44,4 @@
  </plugin>
  </plugins>
  </build>
-</project>
\ No newline at end of file
+</project>
diff --git a/mapsearch/pom.xml b/mapsearch/pom.xml
index 658a211..3f32587 100644
--- a/mapsearch/pom.xml
+++ b/mapsearch/pom.xml
@@ -25,6 +25,13 @@
  <build>
  <plugins>
  <plugin>
+ <artifactId>maven-compiler-plugin</artifactId>
+ <configuration>
+ <source>1.6</source>
+ <target>1.6</target>
+ </configuration>
+ </plugin>
+ <plugin>
  <groupId>org.dstovall</groupId>
  <artifactId>onejar-maven-plugin</artifactId>
  <version>1.4.4</version>


Title: Re: ASM based map locator platform
Post by: prj on October 12, 2012, 01:58:31 PM
Applied.

Odd that it built fine here, I guess I had some Eclipse leftovers in there.


Title: Re: ASM based map locator platform
Post by: phila_dot on October 12, 2012, 04:22:31 PM
I like it. Community projects are good and this can benefit almost everyone.

Trying to think if an IDA script could be written to do most of the dirty work for this...

Do you have a solution for maps that are referenced with a pointer?

KFDMLSDO for example:
mov     r5, mkfanb_w         ; Move Word              
mov     [-r0], r5            ; Move Word              
mov     r2, #613Ch           ; pointer to KFDMLSDO    
movbz   r3, byte_F9F6        ; Move Byte Zero Extend  
shl     r3, #1               ; Shift Left              
add     r2, r3               ; Integer Addition        
mov     r3, [r2]             ; Move Word              
add     r3, #20h ; ' '       ; start of KFDMLSDO z data
mov     [-r0], r3            ; Move Word              
movbz   r2, byte_F9F6        ; Move Byte Zero Extend  
shl     r2, #1               ; Shift Left              
mov     r3, [r2+KFDMLSDO_p]  ; Move Word              
mov     r12, [r3]            ; Move Word              
movbz   r2, byte_F9F6        ; Move Byte Zero Extend  
shl     r2, #1               ; Shift Left              
mov     r13, [r2+KFDMLSDO_p] ; Move Word              
add     r13, #4              ; start of KFDMLSDO x axis
movbz   r2, byte_F9F6        ; Move Byte Zero Extend  
shl     r2, #1               ; Shift Left              
mov     r3, [r2+KFDMLSDO_p]  ; Move Word              
mov     r14, [r3+2]          ; Move Word              
movbz   r2, byte_F9F6        ; Move Byte Zero Extend  
shl     r2, #1               ; Shift Left              
mov     r15, [r2+KFDMLSDO_p] ; Move Word              
add     r15, #14h            ; start of KFDMLSDO y axis

Address 0x81613C contains hex 7C 59. The label KFDMLSDO_p is address 0x81613C.

204h * 4000 + 597C gives 0x81597C, the beginning address for KFDMLSDO.


Title: Re: ASM based map locator platform
Post by: prj on October 12, 2012, 04:58:15 PM
Do you have a solution for maps that are referenced with a pointer?

I will look into it tomorrow.
I don't think it can be done right now - I must amend the XML API to support this.

For now I added the following:
LDRXN
LDRXNZK

Support for "afterRowAxis" and "afterColAxis" as an alternative to "pattern".
This is needed because sometimes the map axis and map come right after another in the code, and there is only one reference (like it is done with LDRXN).

Changes have been pushed to git.


Title: Re: ASM based map locator platform
Post by: masterj on October 13, 2012, 04:24:29 AM
Very very good project indeed! But could I suggest something? Because we have FR that has ME7.5 maps names, could we use these instead? For example CLRHK is CLRSHK. I know that it is me7.5 specific name, but since we have fr file with these names it could be just easier to use search when searching for maps based on documents we have (FR)...


Title: Re: ASM based map locator platform
Post by: masterj on October 13, 2012, 08:37:04 AM
BTW: can someone help me to build latest source? I downloaded maven, created JAVA_HOME variable and it always gives me ERROR\when I try to compile...

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] me7-tools ......................................... SUCCESS [0.016s]
[INFO] ols-parser ........................................ FAILURE [18.045s]
[INFO] xdf-generator ..................................... SKIPPED
[INFO] mapsearch ......................................... SKIPPED
[INFO] maplocator-ui ..................................... SKIPPED
[INFO] maplocator-xmlplugin .............................. SKIPPED

UPDATE: IGNORE THAT :) Wrong JAVA_HOME path ;)


Title: Re: ASM based map locator platform
Post by: prj on October 13, 2012, 04:39:39 PM
Very very good project indeed! But could I suggest something? Because we have FR that has ME7.5 maps names, could we use these instead? For example CLRHK is CLRSHK. I know that it is me7.5 specific name, but since we have fr file with these names it could be just easier to use search when searching for maps based on documents we have (FR)...
Don't think it matters too much to be honest...


Title: Re: ASM based map locator platform
Post by: Rick on October 15, 2012, 01:19:34 AM
Nice work :)


Title: Re: ASM based map locator platform
Post by: prj on October 15, 2012, 05:53:36 AM
I hope someone will contribute some XML's as well ;)

I am looking at the pointer based map access and I don't see a good way to solve this right now.
But remember, there is an option to write drop-in plugins in Java as well.


Title: Re: ASM based map locator platform
Post by: prj on October 17, 2012, 02:10:42 PM
Changelog:
17.10.2012
- Support for multiple patterns
- Better axis length detection
- Added KFZW/KFZW2
- Maps that are found are now sorted alphabetically in the UI

Updated the user guide and updated the binary in the first post.
Also pushed changes to GIT.


Title: Re: ASM based map locator platform
Post by: prj on October 17, 2012, 03:08:09 PM
Added KFMIOP, MLOFS.
https://github.com/prj/me7-tools/tree/master/maplocator-ui/resources/me7xmls (https://github.com/prj/me7-tools/tree/master/maplocator-ui/resources/me7xmls)


Title: Re: ASM based map locator platform
Post by: rob.mwpropane on October 17, 2012, 03:10:05 PM
Time to give back to the forum.


Not to kiss ass, but you have been;) Thanks for this and all the hard work...


Title: Re: ASM based map locator platform
Post by: prj on October 17, 2012, 03:40:38 PM
I just hope someone besides me starts helping out with the XML's.
Otherwise it's a doomed project ;)

A good start would be to find all the emissions based things. They are pretty simple to write XML's for, as it's just single value.


Title: Re: ASM based map locator platform
Post by: nyet on October 17, 2012, 03:44:09 PM
prj: i have a lot of incentive to help out, just bad timing; super busy with other stuff lately.

in particular, a complete set of xmls combined with an ASAP2 writer would be the most amazing thing ever.


Title: Re: ASM based map locator platform
Post by: nyet on October 17, 2012, 03:46:01 PM
btw,  something equivalent for ram locations would be freaking AMAZING


Title: Re: ASM based map locator platform
Post by: prj on October 18, 2012, 12:00:49 AM
btw,  something equivalent for ram locations would be freaking AMAZING

This can easily be done using the current program and some XML's.


Title: Re: ASM based map locator platform
Post by: prj on November 01, 2012, 07:33:05 AM
Added:
CDKAT
CDSLS
CWDLSAHK
TVUB


Title: Re: ASM based map locator platform
Post by: prj on November 01, 2012, 07:49:47 AM
Current full list of signatures:
CATR
CDKAT
CDSLS
CLRHK
CWDLSAHK
KFLBTS
KFMIOP
KFZW
KFZW2
KRKTE
KRKTE_8
LAMFA
LAMFA_COLAXIS
LAMFA_ROWAXIS
LDRXN
LDRXNZK
LDRXNZK_ROWAXIS
LDRXN_ROWAXIS
MLHFM
MLOFS
SNM16GKUB
SNM16OPUW
SNM16ZUUB
SRL11OPUW
SRL12GKUB
SRL12GKUW
SRL12ZUUB
TVUB
TVUB_AXIS
VARDEF

They are not of course all detected in all files, but it still works for many :)


Title: Re: ASM based map locator platform
Post by: masterj on November 01, 2012, 08:01:55 AM
Current full list of signatures:
CATR
CDKAT
CDSLS
CLRHK
CWDLSAHK
KFLBTS
KFMIOP
KFZW
KFZW2
KRKTE
KRKTE_8
LAMFA
LAMFA_COLAXIS
LAMFA_ROWAXIS
LDRXN
LDRXNZK
LDRXNZK_ROWAXIS
LDRXN_ROWAXIS
MLHFM
MLOFS
SNM16GKUB
SNM16OPUW
SNM16ZUUB
SRL11OPUW
SRL12GKUB
SRL12GKUW
SRL12ZUUB
TVUB
TVUB_AXIS
VARDEF

They are not of course all detected in all files, but it still works for many :)

Nice:) Gonna build your latest version... :)


Title: Re: ASM based map locator platform
Post by: prj on November 01, 2012, 10:06:31 AM
You can just pull the XML's and copy paste them into the version that is built.
I have not changed the code really, only added new XML's.


Title: Re: ASM based map locator platform
Post by: Bische on November 01, 2012, 12:14:44 PM
Thanks for sharing this.

One day I will learn to row around the code as you guys do, and contribute back.

Thing is, I dont know where to start reading or what to start practice with. Do I need to learn how to build stuff and then learn to dissassemble it? I feel I just need to get the ball rolling, I dont want to be spoon feed, I just want to learn..


Title: Re: ASM based map locator platform
Post by: prj on November 01, 2012, 02:35:14 PM
Get IDA pro and the C166 users manual... and then go from there.
I've been reverse engineering various things for over 10 years now, it's definitely not a quick process to learn this.


Title: Re: ASM based map locator platform
Post by: Bische on November 01, 2012, 08:16:21 PM
Get IDA pro and the C166 users manual... and then go from there.
I've been reverse engineering various things for over 10 years now, it's definitely not a quick process to learn this.


I have IDApro already. C166 is the name of the processor inside our ECU's right? :)

I have figured im not going to learn in a couple of months, but I must start somewhere.

Thanks prj


Title: Re: ASM based map locator platform
Post by: masterj on November 12, 2012, 04:03:59 PM
Just wanted to report results on my file :)

Those were incorrectly found:
KFZW
LAMFA
LAMFA_COLAXIS
LAMFA_ROWAXIS
VARDEF


My ori bin is here: http://nefariousmotorsports.com/forum/index.php?action=dlattach;topic=1193.0;attach=4329 (http://nefariousmotorsports.com/forum/index.php?action=dlattach;topic=1193.0;attach=4329)


Title: Re: ASM based map locator platform
Post by: amd is the best on November 12, 2012, 05:15:59 PM
Phenomenal work and rep has been given.

Will test this out tonight with a few files.


Title: Re: ASM based map locator platform
Post by: prj on November 13, 2012, 03:58:07 AM
Just wanted to report results on my file :)

Those were incorrectly found:
KFZW
LAMFA
LAMFA_COLAXIS
LAMFA_ROWAXIS
VARDEF


My ori bin is here: http://nefariousmotorsports.com/forum/index.php?action=dlattach;topic=1193.0;attach=4329 (http://nefariousmotorsports.com/forum/index.php?action=dlattach;topic=1193.0;attach=4329)

Yes, sometimes happens on a few files - correct the patterns and post them here.
Sorry, but I want to make it pretty clear, I am not going to go through every file where it does not find something correctly.

The platform is completely open and open source. If you want to fix some .xml, you can do it.
The only bugs I am interested in are bugs in the java code.


Title: Re: ASM based map locator platform
Post by: masterj on November 13, 2012, 04:37:28 AM
Yes, sometimes happens on a few files - correct the patterns and post them here.
Sorry, but I want to make it pretty clear, I am not going to go through every file where it does not find something correctly.

The platform is completely open and open source. If you want to fix some .xml, you can do it.
The only bugs I am interested in are bugs in the java code.

Yes, I know the idea of this locator :) Just wanted to say that people wouldn't blindly believe every map it finds :) Although it is correct for most part...

BTW: I could fix if I knew how to do the asm :) all i do is compare binaries between known and unknown in winols


Title: Re: ASM based map locator platform
Post by: masterj on November 19, 2012, 03:55:06 PM
Just created my first definition for KAMFZ map. Problem is I can't push it to github. Is it private depository?


Title: Re: ASM based map locator platform
Post by: phila_dot on November 19, 2012, 04:24:30 PM
Just wanted to report results on my file :)

Those were incorrectly found:
KFZW
LAMFA
LAMFA_COLAXIS
LAMFA_ROWAXIS
VARDEF


My ori bin is here: http://nefariousmotorsports.com/forum/index.php?action=dlattach;topic=1193.0;attach=4329 (http://nefariousmotorsports.com/forum/index.php?action=dlattach;topic=1193.0;attach=4329)

If you get the chance you could compare these in your file against prj's definition and see if you can adapt it to work with your file and still be unique enough.


Title: Re: ASM based map locator platform
Post by: nyet on November 19, 2012, 05:24:39 PM
Just created my first definition for KAMFZ map. Problem is I can't push it to github. Is it private depository?

just make a clone on github and do a pull request.


Title: Re: ASM based map locator platform
Post by: masterj on November 19, 2012, 07:25:12 PM
just make a clone on github and do a pull request.

Well I'm not that good with github, just using github for windows gui.

Anyway, here's my own definitions for some of the PROKONAL maps. (Should work with all ME7.5)

Only problematic map is CDBKVP, which sometimes isn't correctly defined, but at the moment I do not know how to fix it


Title: Re: ASM based map locator platform
Post by: prj on November 20, 2012, 02:54:36 AM
You have masked out too little.
These XML's will probably just work for one binary (yours) and not any other binaries.

The registers do not have to be the same between binaries and so on.

Have you tested this on any other ME7.5 binaries besides your own?
Have you tested with 29F400 ECU's?

P.S.
Your submission is appreciated, I just want to make sure it is useful for things beyond one binary.
Also correct way to submit these is a push request on github. But I can add them manually, just a bit of extra work.


Title: Re: ASM based map locator platform
Post by: masterj on November 20, 2012, 05:19:45 AM
You have masked out too little.
These XML's will probably just work for one binary (yours) and not any other binaries.

The registers do not have to be the same between binaries and so on.

Have you tested this on any other ME7.5 binaries besides your own?
Have you tested with 29F400 ECU's?

P.S.
Your submission is appreciated, I just want to make sure it is useful for things beyond one binary.
Also correct way to submit these is a push request on github. But I can add them manually, just a bit of extra work.

I have tried these on two binaries, both ME7.5 and both 29F800. Also I have tried to mask more things, but found out that for most prokonal maps this is what you have to use to be 100% sure (atleast on said ecus):
E6 F4 ii ii 64 F4 XX 8B D7 40 06 02 C2 F4 MMXX 01 68 41 2D 05 E6 F4 ii ii 74 F4 XX 8B 0D 04

Where ii <- are most important bits, that change accordingly to prokonal map

Of course you could try ii ii XX XX XX XX D7 40 06 02 C2 F4 MMXX XX XX XX XX XX E6 F4 ii ii but for these two ecus they were all the same and for other ecu that i have looked it was completely different pattern

I suspect that we will need to actually define two different versions for each ecu type... Would be good if locator could automatically choose the right one based on ecu where they are different, like CDAGR_f400, CDAGR_f800 if this is really different between these two


Title: Re: ASM based map locator platform
Post by: prj on November 20, 2012, 06:17:45 AM
There is currently already support for adding multiple patterns to a single XML.
All that has be done is make sure that you do not get false positives.

You can add many <pattern> tags into one XML.


Title: Re: ASM based map locator platform
Post by: masterj on November 20, 2012, 11:06:57 AM
There is currently already support for adding multiple patterns to a single XML.
All that has be done is make sure that you do not get false positives.

You can add many <pattern> tags into one XML.

ok, now another question is this:
for example CDHSH and CDHSHE both patterns are almost exactly same except for byte which shows address of actual map. Can the locator skip address that was previously assigned to some map buy definition?

Also CDHSVE vs CDKAT problem!


Title: Re: ASM based map locator platform
Post by: phila_dot on November 20, 2012, 12:09:19 PM
This may be a little too much, but if it could find potential target in pattern then read RAM address from hex to x-ref for reverse lookup or further validation. If that makes sense.

B_cdhsh gets referenced in xxx1 pattern, read RAM address from pattern, x-ref other references to B_cdhsh in xxx2 pattern for further validation possibly, x-ref to B_cdhsh set by CDHSH in xxx3 pattern.

Just thinking out loud.


Title: Re: ASM based map locator platform
Post by: prj on November 20, 2012, 12:36:11 PM
If you come up with a way to write this into XML, sure.


Title: Re: ASM based map locator platform
Post by: prj on November 20, 2012, 12:37:23 PM
ok, now another question is this:
for example CDHSH and CDHSHE both patterns are almost exactly same except for byte which shows address of actual map. Can the locator skip address that was previously assigned to some map buy definition?
It does not skip right now, but this can be added, as it makes sense.
The problem is which comes first.

So I think you have to find some way to differentiate them in the code.


Title: Re: ASM based map locator platform
Post by: masterj on November 20, 2012, 12:47:21 PM
It does not skip right now, but this can be added, as it makes sense.
The problem is which comes first.

So I think you have to find some way to differentiate them in the code.

today i've worked on bunch of prokonal maps (sent pull request on github). I absolutely had to update CDSLS because it gave me false address for map (read notes of commit).


Title: Re: ASM based map locator platform
Post by: prj on November 20, 2012, 12:47:39 PM
Saw the pull request, thank you.

I am happy to improve the pattern matcher and the XML format - but I need some exact ideas of how it could look in the XML.
After that I can update the XML plugin to parse the new format XML :)


Title: Re: ASM based map locator platform
Post by: prj on November 20, 2012, 12:49:31 PM
One thing you can do about false positives is put the patterns in the same file.
If the first pattern is not found only then will the second be evaluated.

If the first (more exact) pattern matches then the second will not be evaluated and there will be no false positive.


Title: Re: ASM based map locator platform
Post by: masterj on November 20, 2012, 03:57:26 PM
I have one idea:
What if locator could work on unlimited ammount of "jump-levels", so we didn't have to search only one specific place for pattern but multiple that are connected to each other, like:
le'ts say we need to map address @ 18000
and we find xref to this address @ 25000 in some kind of pattern like: hh hh hh XX hh hh MM hh hh XX hh, where MM is reference to 18000. But let's say we have two similar looking addresses with same pattern
25000: hh hh hh XX hh hh MM hh hh XX hh
26000: hh hh hh XX hh hh MM hh hh XX hh

Now there's no way to make definitions for 18000 map, because we don't have unique pattern in this situation. But if we let's say have:
25000: R1R1 hh hh hh XX hh hh MM hh hh XX hh
26000: R2R2 hh hh hh XX hh hh MM hh hh XX hh

where R1R1 and R2R2 are two different refs to these address, then we could make our pattern to something like these:
55000: hh xx hh xx xx hh MM xx hh hh | R1R1 hh hh hh XX hh hh MM hh hh XX hh
56000: hh hh xx xx hh hh xx hh MM xx hh hh | R2R2 hh hh hh XX hh hh MM hh hh XX hh

And that would mean:
we search 55000 by pattern: hh xx hh xx xx hh MM xx hh hh, where MM are the part with the ref to diff address. symbol "|" would mean secondary pattern that was referenced from first to address 25000 and here we search again R1R1 hh hh hh XX hh hh MM hh hh XX hh for MM which finally helps us define our map. Of course for this to work we might need not two but few different levels of refs. I think that would be possible in the current stage of locator, no?

PS> I think Philla suggested same thing?


Title: Re: ASM based map locator platform
Post by: prj on November 21, 2012, 06:36:07 AM
You mean that the | symbol would mean that it first finds the first pattern and then searches for the second pattern right after the first?
Yes, this is possible, I can add this.

I am not sure I quite understand this whole ref stuff you are trying to get at though.


Title: Re: ASM based map locator platform
Post by: masterj on November 21, 2012, 01:07:25 PM
Could you accept my pull request? Im ready to pull more maps but can't until my previous pull request is accepted ;)

P.S> I meant multi pattern at different address locations that are connected to each other by CODE XREF or DATA XREF...


Title: Re: ASM based map locator platform
Post by: prj on November 21, 2012, 04:03:02 PM
Could you accept my pull request? Im ready to pull more maps but can't until my previous pull request is accepted ;)
I can't accept your pull request because that would blow up CDSLS.
Can you modify CDSLS.xml so that you have both patterns in there? Perhaps your pattern as the first one?
You can see the problem here:
https://github.com/masterjguscius/me7-tools/commit/ac319f942b85c753c553686ddd2596e3d95e4581 (https://github.com/masterjguscius/me7-tools/commit/ac319f942b85c753c553686ddd2596e3d95e4581)

Quote
P.S> I meant multi pattern at different address locations that are connected to each other by CODE XREF or DATA XREF...
I think this is a little too complex. Are you sure this is needed?
You can easily skip large chunks with XX{number}.

As for references - this is just a binary matcher. It does not understand *code* per se.
Do you want to be able to refer to a location that has been already found in the pattern matcher ?


Title: Re: ASM based map locator platform
Post by: masterj on November 21, 2012, 04:46:51 PM
I can't accept your pull request because that would blow up CDSLS.
Can you modify CDSLS.xml so that you have both patterns in there? Perhaps your pattern as the first one?
You can see the problem here:
https://github.com/masterjguscius/me7-tools/commit/ac319f942b85c753c553686ddd2596e3d95e4581 (https://github.com/masterjguscius/me7-tools/commit/ac319f942b85c753c553686ddd2596e3d95e4581)
I think this is a little too complex. Are you sure this is needed?
You can easily skip large chunks with XX{number}.

As for references - this is just a binary matcher. It does not understand *code* per se.
Do you want to be able to refer to a location that has been already found in the pattern matcher ?

i have repared a new commit with fixed SLS, but I do not know how to edit pull request :/ can you just discard it and then i will make a new one with everything packed


Title: Re: ASM based map locator platform
Post by: phila_dot on November 21, 2012, 05:13:12 PM
I can't accept your pull request because that would blow up CDSLS.
Can you modify CDSLS.xml so that you have both patterns in there? Perhaps your pattern as the first one?
You can see the problem here:
https://github.com/masterjguscius/me7-tools/commit/ac319f942b85c753c553686ddd2596e3d95e4581 (https://github.com/masterjguscius/me7-tools/commit/ac319f942b85c753c553686ddd2596e3d95e4581)
I think this is a little too complex. Are you sure this is needed?
You can easily skip large chunks with XX{number}.

As for references - this is just a binary matcher. It does not understand *code* per se.
Do you want to be able to refer to a location that has been already found in the pattern matcher ?

The only thing unique for the PROKON codewords is the word variable and the bit value, generally.

My suggestion was to use a pattern to locate the bit (B_cdxxx), then search for the codeword pattern and match RAM address/ bit value to confirm codeword. Or something along those lines. I haven't gotten the chance to play with your utility yet, so I don't really know if this is feasible.


Title: Re: ASM based map locator platform
Post by: nyet on November 21, 2012, 06:32:12 PM
i have repared a new commit with fixed SLS, but I do not know how to edit pull request :/ can you just discard it and then i will make a new one with everything packed

you can cancel the pull request and issue another one

If possible, since nobody else is using your git tree, you might consider a rebase as well to hide whatever thrashing you did in the commit history.


Title: Re: ASM based map locator platform
Post by: prj on November 22, 2012, 02:02:13 AM
The only thing unique for the PROKON codewords is the word variable and the bit value, generally.

My suggestion was to use a pattern to locate the bit (B_cdxxx), then search for the codeword pattern and match RAM address/ bit value to confirm codeword. Or something along those lines. I haven't gotten the chance to play with your utility yet, so I don't really know if this is feasible.

Ok, so you want to locate a certain location, and then use the address as part of another pattern?
So let's say you find B_CDXXX, say you get two bytes 0x5467 (random pick), and you want to include these in the main pattern literally?
This could be done.


Title: Re: ASM based map locator platform
Post by: prj on November 22, 2012, 05:52:02 AM
I have pushed a new version.

The lookup of patterns is now multithreaded.
This is the operation that always takes the longest, and it will get worse as more patterns are added.

Now all patterns are pre-cached via a thread pool, and already there is a small performance increase.


Title: Re: ASM based map locator platform
Post by: dream3R on October 07, 2013, 04:52:08 AM
I'd like to contribute to the XML library.

prj do you have the screens from the original xml explanation post?  I'm fairly new to IDA and those will help me understand the process.



Title: Re: ASM based map locator platform
Post by: prj on October 09, 2013, 01:37:13 AM
I don't think I have them unfortunately :(


Title: Re: ASM based map locator platform
Post by: dream3R on October 13, 2013, 02:10:15 AM
Hi Prj,

That's a pity.


Title: Re: ASM based map locator platform
Post by: dream3R on December 01, 2013, 02:33:16 PM
Hello prj.  Progress has been made.

The offsets on Volvo are different.  Map address is not offset by 0x80000 like in vag, it's 0.

Anyway, is there a quick hack to tell the software not to apply the vag offset?  I'm hoping to avoid java/maven witchcraft :)


Title: Re: ASM based map locator platform
Post by: ddillenger on December 01, 2013, 06:10:47 PM
CDHSH XML attached. HSHE is right after HSH in pretty much every binary, so 2 for 1 here.


Title: Re: ASM based map locator platform
Post by: dream3R on December 02, 2013, 02:18:00 AM
CDHSH XML attached. HSHE is right after HSH in pretty much every binary, so 2 for 1 here.

Hi,

Thanks for taking the time but you got the wrong end of the stick, probably my explanation.  Volvo's bin's do not need offset like VAG so the calcs are wrong on Volvo bins.

Stunning software though prj, well done!!


Title: Re: ASM based map locator platform
Post by: ddillenger on December 02, 2013, 05:27:46 AM
That was for VAG. You bumping the thread just make me think to post it.


Title: Re: ASM based map locator platform
Post by: dream3R on December 02, 2013, 06:02:27 AM
That was for VAG. You bumping the thread just make me think to post it.

Ah sorry.

You might want to commit that to the github.


Title: Re: ASM based map locator platform
Post by: ddillenger on December 02, 2013, 06:21:44 AM
dream3r:

I have something that might help you. It works by referencing a library of files against one you have chosen (and provided the address for the map/variable you are searching), and looking for a pattern that applies to ALL of them. It's cmd line of course. You're going to have to email me with some similar binaries and a known address so I can verify it'll work for volvo.


Title: Re: ASM based map locator platform
Post by: dream3R on December 02, 2013, 06:26:02 AM
dream3r:

I have something that might help you. It works by referencing a library of files against one you have chosen (and provided the address for the map/variable you are searching), and looking for a pattern that applies to ALL of them. It's cmd line of course. You're going to have to email me with some similar binaries and a known address so I can verify it'll work for volvo.

That sounds interesting!  I'm going to give compiling this with changes a shot, it looks like it will be able to report memory locations on different bins, which is really what I was hoping it could do, for the logger.  Why recreate the wheel and all that ...

I've got to grips with assembly, thankfully, and am in the process of defining my file and creating a RAM map with IDA Pro.  There's some interesting bits of code for sure.

The really good news is that I think TunnerPro RT is interoperable with my Microcontrller code, which is amazing if it works.  I wrote a basic Rs232 interface for controlling it from Windows and using that it should work...

I'm also working on a J2534 version when time allows.


Title: Re: ASM based map locator platform
Post by: guitar24t on December 02, 2013, 09:50:42 AM
I created a pull request for an update to the Xml plugin to allow correct operation on Volvo binaries.
I've attached the source change here as well, for anyone who wants to compile it. The more testing, the better.

Oh, and I should note that the XML files need to be renamed to .audi.xml and ones for volvo will be named .volvo.xml.
I updated the pull to reflect those changes.


Title: Re: ASM based map locator platform
Post by: guitar24t on December 02, 2013, 11:27:28 AM
I created a pull request for an update to the Xml plugin to allow correct operation on Volvo binaries.
I've attached the source change here as well, for anyone who wants to compile it. The more testing, the better.

Oh, and I should note that the XML files need to be renamed to .audi.xml and ones for volvo will be named .volvo.xml.
I updated the pull to reflect those changes.

Here's a compiled version of the software with these changes integrated.


Title: Re: ASM based map locator platform
Post by: prj on December 04, 2013, 09:25:37 AM
I need to accept the request, been so busy...
If you want straight out access to the repository as a contributor let me know and I'll add you.


Title: Re: ASM based map locator platform
Post by: dream3R on December 04, 2013, 09:32:09 AM
Here is my finding for using it for RAM variables in Volvo ME7:

I calculated:  DPP * 4000 + offset - 8000 = ram value for the 0x300000 -> 0x301000 range and DPP = 0 for 0xC000 -> D7FF and no offset.

Not sure if that's right, but it was picking up miopt_w, nmot and another I can't remember using the above.



Title: Re: ASM based map locator platform
Post by: guitar24t on December 06, 2013, 10:31:30 AM
I need to accept the request, been so busy...
If you want straight out access to the repository as a contributor let me know and I'll add you.
I wouldn't mind being a contributor to that repo. That way if I need to tweak the Volvo memory calcs a little, I don't have to make more pull requests. ;)


Title: Re: ASM based map locator platform
Post by: prj on December 06, 2013, 02:47:40 PM
Done.

dream3r, let me know if you want access too.


Title: Re: ASM based map locator platform
Post by: aef on January 22, 2014, 06:02:03 AM
great tool!

looking forward to .kp export


Title: Re: ASM based map locator platform
Post by: aef on February 15, 2014, 04:03:22 PM
Looks like XDF export didnt work.

All Tables are empty in Tunerpro. Tried to compare with a working xdf v1.50 but had no luck.

Attached my file, the created non working xdf and the working xdf from the board.



Title: Re: ASM based map locator platform
Post by: prj on February 15, 2014, 04:10:23 PM
Used to work fine before. I don't know what changed.


Title: Re: ASM based map locator platform
Post by: guitar24t on March 07, 2014, 10:12:21 AM
Looks like XDF export didnt work.

All Tables are empty in Tunerpro. Tried to compare with a working xdf v1.50 but had no luck.

Attached my file, the created non working xdf and the working xdf from the board.



For whatever reason, your XDF file contains commas instead of dots in the equations. TunerPro does not like that.
This is the XDF file that was generated from your bin on my computer (with the latest version of map locator).


Title: Re: ASM based map locator platform
Post by: nyet on March 07, 2014, 10:14:57 AM
For whatever reason, your XDF file contains commas instead of dots in the equations.

Very annoying locale issues :/


Title: Re: ASM based map locator platform
Post by: mmedeiros on September 04, 2014, 05:00:19 PM
This is great everyone, I want to trust to develop a plug-in for the MED17......!


Title: Re: ASM based map locator platform
Post by: nubcake on October 15, 2016, 01:41:28 PM
I think this tool is way underrated.
If someone were to add "multipass" parsing - it could become even more awesome. On the other hand, it then would become more of a "professional" tool.
Anyway, I played some with it and made a bunch of other XMLs. They are not written in the most portable way and can be inaccurate sometimes, especially with codeword (CDxxx) constants, but here they are anyway.


Title: Re: ASM based map locator platform
Post by: nyet on October 15, 2016, 02:35:29 PM
I think this tool is way underrated.

I think it isn't underrrated, but it is VERY much underutilized... particularly in that it can be extended to find RAM locations as well!

Thanks for taking the time to develop additional xml definitions.


Title: Re: ASM based map locator platform
Post by: prj on October 17, 2016, 09:49:29 AM
Well, it's 4 years old now :)
If more people worked on patterns it would be quite useful to almost instantly define most needed maps in any me7.

If you want to add anything - feel free to. It's open source.
I don't have any time for stuff like this these days :(


Title: Re: ASM based map locator platform
Post by: nubcake on June 13, 2017, 05:04:30 PM
Seen someone ask today for mibas_w, misolv_w and mizsolv_w. So I thought that instead of helping just one person, I'll help a bunch - and try the old maplocator for memory vars. Well, it looks like the DPP algo is a bit fucked for those. I didn't want to spend much time (let alone dig in the source), so I came up with this hack instead.

It will tell you that the memory address is 0x3F781234 (for example). Looks like total bullshit, but if you just pretend that F7 is not there (and note it as 0x381234 instead) - it'll be correct! Tested it on couple of binaries and it seems to be right.

As told previously, this tool is way under-utilized indeed. :)



Title: Re: ASM based map locator platform
Post by: derbali01 on October 12, 2017, 05:19:59 PM
thxxx a lots


Title: Re: ASM based map locator platform
Post by: quattroGmbH on December 19, 2017, 05:12:44 PM
Does anyone know how to resolve this compile error? Do I need to set a new override in xdfproject.java? I’ll post the stack trace and debug tonight.

[ERROR] /C:/me7-tools-master/maplocator-ui/src/main/java/com/prj/tuning/maplocator/export/xdf/model/Project.java:[17,8]com.prj.tuning.maplocator.export.xdf.model.Project is not abstract and does not override abstract method getConstants() in com.prj.tuning.xdf.binding.XdfProject

I changed the plug in in mapsearch to com.jolira as org.dstovall was giving me plug in errors

FYI anyone using this you will have to remove reference to the google cache in the mapsearch POM as the cache is permanently offline and the plugin is now incorporated in Maven. Using maven 3.5.2, jdk 1.8 and windows 7 pro.

Thank you!


Title: Re: ASM based map locator platform
Post by: quattroGmbH on January 03, 2018, 09:05:20 AM
After a lot of faffing around with trying to add overrides to the project.java file I found that the they way java overrides the abstract method is different in java 6 to 7. Since Maven 3 wont run with an older version of java, I reverted back to the original java 6 and maven 2.2.1.

There was another compile error due to the maplocator-me7plugin module being listed in the parent POM but doesnt have a supporting child POM. Since Im no programmer and not writing POMs removing reference to the maplocator-me7plugin module let me compile the project.

PRJ, If you are still active on nef, how do I utilize the mapsearch function? I dont see the jar listed in the maplocator-release lib and adding the SNAPSHOT version from mapsearch\target to the release lib does not change the search results. Id really like to just auto scan a couple binaries.

Thanks prj!


Title: Re: ASM based map locator platform
Post by: clubman on January 03, 2018, 09:34:27 AM
I tried the tool today but I get:
Quote
Exception in thread 'main' java.lang.ClassCastException: java.base/jdk.internal.loader.ClassLoaders$AppClassLoader cannot cast to java.base/java.net.URLClassLoader
at com.prj.tuning.maplocator.ui.Loader.main(Loader.java:15)
at com.prj.tuning.maplocator.ui.CrossPlatformRunner.main(CrossPlatformRunner.java:31)

Anyone knows how to fix it?


Title: Re: ASM based map locator platform
Post by: quattroGmbH on January 03, 2018, 03:00:32 PM
Looks like a java error. What’s your platform, java version?


Title: Re: ASM based map locator platform
Post by: nyet on January 03, 2018, 03:46:45 PM
Working on it now. Fixed maven issues and missing pom, now looking at compile error

https://github.com/nyetwurk/me7-tools


Title: Re: ASM based map locator platform
Post by: nyet on January 03, 2018, 04:12:04 PM
My fork should compile now.


Title: Re: ASM based map locator platform
Post by: quattroGmbH on January 03, 2018, 06:06:59 PM
Awesome Nye! It seems like compiling this project should incorporate the auto generate mapsearch function from this thread? http://nefariousmotorsports.com/forum/index.php?topic=2723.0

I’m mostly trying to get the mapsearch function to auto scan. I was thinking this project needed to compile it all together. It also looks like the export to kp function is greyed out after analyzing a bin. With all the different xmls I can’t get about 10 maps and axis. Better than nothing but not quite up to the m and k auto xdfs.


Title: Re: ASM based map locator platform
Post by: clubman on January 04, 2018, 02:32:35 AM
Hello,

Thank you for your fast reply. Unfortunately I am not a Java expert and have tried to fork it with maven and I get the cannot find javac error. I have set the JAVA_HOME environment variable but still it points to JRE instead of JDK and cannot compile. Can you please update the 1st post with a release that is now able to run?

Thank you!


Title: Re: ASM based map locator platform
Post by: nyet on January 04, 2018, 11:50:19 AM
Hello,

Thank you for your fast reply. Unfortunately I am not a Java expert and have tried to fork it with maven and I get the cannot find javac error. I have set the JAVA_HOME environment variable but still it points to JRE instead of JDK and cannot compile. Can you please update the 1st post with a release that is now able to run?

Thank you!

check PATH


Title: Re: ASM based map locator platform
Post by: quattroGmbH on January 04, 2018, 11:58:38 AM
Hello,

Thank you for your fast reply. Unfortunately I am not a Java expert and have tried to fork it with maven and I get the cannot find javac error. I have set the JAVA_HOME environment variable but still it points to JRE instead of JDK and cannot compile. Can you please update the 1st post with a release that is now able to run?

Thank you!

https://docs.oracle.com/cd/E19182-01/820-7851/inst_cli_jdk_javahome_t/


Title: Re: ASM based map locator platform
Post by: quattroGmbH on January 04, 2018, 12:00:35 PM
After the updates I went back to maven 3.5.2 and JDK 9. I’m still getting a complie error. At work now so I’ll update with the new error tonight.


Title: Re: ASM based map locator platform
Post by: clubman on January 04, 2018, 03:27:00 PM
https://docs.oracle.com/cd/E19182-01/820-7851/inst_cli_jdk_javahome_t/

Yeap that's what I 've done.. Also using JDK9..


Title: Re: ASM based map locator platform
Post by: nyet on January 04, 2018, 03:32:36 PM
Using jdk9 and mvn 3.5

no issues

$ mvn --version
Apache Maven 3.5.0
Maven home: /usr/share/maven
Java version: 9.0.1, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-9-openjdk-amd64
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "4.14.0-2-amd64", arch: "amd64", family: "unix"


Title: Re: ASM based map locator platform
Post by: clubman on January 04, 2018, 04:24:59 PM
OK now I tried it on my other laptop and I get other error:

It says BUILD FAILURE on maplocator-ui because it cannot find org.apache.maven.plugins:maven-jar-plugin:jar:3.7.0 in https://repo.maven.apache.org/maven2 and something about updates..


Title: Re: ASM based map locator platform
Post by: quattroGmbH on January 04, 2018, 07:48:17 PM
OK now I tried it on my other laptop and I get other error:

It says BUILD FAILURE on maplocator-ui because it cannot find org.apache.maven.plugins:maven-jar-plugin:jar:3.7.0 in https://repo.maven.apache.org/maven2 and something about updates..

Looks like we need the Linux upgrade :)

I get the same error
[INFO] Building maplocator-ui 0.0.1-SNAPSHOT
[INFO] ------------------------------------------------------------------------
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-jar-plugin/3.7.0/maven-jar-plugin-3.7.0.pom
[WARNING] The POM for org.apache.maven.plugins:maven-jar-plugin:jar:3.7.0 is missing, no dependency information available
Downloading from central: https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-jar-plugin/3.7.0/maven-jar-plugin-3.7.0.jar
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] me7-tools .......................................... SUCCESS [  0.000 s]
[INFO] ols-parser ......................................... SUCCESS [ 26.583 s]
[INFO] xdf-generator ...................................... SUCCESS [  2.559 s]
[INFO] mapsearch .......................................... SUCCESS [ 16.660 s]
[INFO] maplocator-ui ...................................... FAILURE [  0.156 s]
[INFO] maplocator-xmlplugin ............................... SKIPPED
[INFO] maplocator-me7plugin ............................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 46.286 s
[INFO] Finished at: 2018-01-04T18:45:21-08:00
[INFO] Final Memory: 19M/65M
[INFO] ------------------------------------------------------------------------
[ERROR] Plugin org.apache.maven.plugins:maven-jar-plugin:3.7.0 or one of its dependencies could not be resolved: Could not find artifact org.apache.maven.plugins:maven-jar-plugin:jar:3.7.0 in central (https://repo.maven.apache.org/maven2) -> [Help 1]


$ mvn -version
Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T00:58:13-07:00)
Maven home: C:\Program Files\Apache\maven\apache-maven-3.5.2
Java version: 9.0.1, vendor: Oracle Corporation
Java home: C:\Program Files\Java\jdk-9.0.1
Default locale: en_US, platform encoding: Cp1252
OS name: "windows 7", version: "6.1", arch: "amd64", family: "windows"


Title: Re: ASM based map locator platform
Post by: quattroGmbH on January 04, 2018, 07:54:39 PM
Yeap that's what I 've done.. Also using JDK9..

Did you set Java and Maven system variables too?


Title: Re: ASM based map locator platform
Post by: nyet on January 04, 2018, 08:29:55 PM
try again, i did something wrong hehe


Title: Re: ASM based map locator platform
Post by: quattroGmbH on January 04, 2018, 08:48:27 PM
try again, i did something wrong hehe

Great success! Thank you Nye! I really appreciate you supporting this, even 5 years later. :) :)


Title: Re: ASM based map locator platform
Post by: quattroGmbH on January 04, 2018, 08:59:31 PM
C:\me7-tools\maplocator-ui\maplocator-release>java -jar maplocator-ui.jar
Exception in thread "main" java.lang.ClassCastException: java.base/jdk.internal.
loader.ClassLoaders$AppClassLoader cannot be cast to java.base/java.net.URLClass
Loader
        at com.prj.tuning.maplocator.ui.Loader.main(Loader.java:15)
        at com.prj.tuning.maplocator.CrossPlatformRunner.main(CrossPlatformRunne
r.java:31)

C:\me7-tools\maplocator-ui\maplocator-release>

Got this error and jar wouldnt open under Java 9. Downgraded to Java 8 and it opens right up.
 


Title: Re: ASM based map locator platform
Post by: nyet on January 04, 2018, 10:59:25 PM
I see that too. Working on it now.


Title: Re: ASM based map locator platform
Post by: nyet on January 04, 2018, 11:04:16 PM
Urgh. java9 broke classLoader big time. This could take a bit of effort.


Title: Re: ASM based map locator platform
Post by: nyet on January 04, 2018, 11:56:35 PM
Yea, I am likely not going to be able to fix this without prj's help


Title: Re: ASM based map locator platform
Post by: prj on January 05, 2018, 12:17:52 AM
I saw your pull requests for the me7-tools stuff too, going to look at it over the weekend.


Title: Re: ASM based map locator platform
Post by: clubman on January 05, 2018, 02:11:11 AM
quattroGmbH, which version of jdk8 did you install?

I installed  Java SE Development Kit 8u151 but still it is not working for me. We get that same error about the artifact.


Title: Re: ASM based map locator platform
Post by: quattroGmbH on January 05, 2018, 08:28:11 AM
quattroGmbH, which version of jdk8 did you install?

I installed  Java SE Development Kit 8u151 but still it is not working for me. We get that same error about the artifact.

Did you download the newest version? Nye fixed that error yesterday. I compiled with 9, and then opened with 1.8.151


Title: Re: ASM based map locator platform
Post by: clubman on January 05, 2018, 11:23:56 AM
Yeah I compiled it with9 but I get the same error with 151 and 152. Do I need to run it on the jdk or jre directory (or add them to the system variables)?

The thing is that I can't change the Java version on the PC. Whatever I do (change path, java_version etc etc) it still shows version 9.01+11 using java -version.

Edit: Ok I just uninstalled version 9 and it ran. Only found 3 axis and KFLBTS though on my Volvo ME7 file. Can I do something to help it locate more maps or thats it?


Title: Re: ASM based map locator platform
Post by: quattroGmbH on January 05, 2018, 04:21:28 PM
Volvo support is limited. You would have to create more Volvo specific XMLs. Do you have any similar, more defined binaries? There is a findmap 0.3 floating around. Not sure if it will work with Volvo. Worth giving it a shot


Title: Re: ASM based map locator platform
Post by: clubman on January 06, 2018, 01:38:40 AM
I’ll search for it,thanks.. I have managed to define around 35 maps on my bin manually and using friends’ help. How do I define an XML? Is there a manual? The pattern, where do i get it from? Are those values like contained always in a map and that’s how it identifies them? I checked some of the patterns but they are not incremental rather than mixed values..

Thanks!


Title: Re: ASM based map locator platform
Post by: quattroGmbH on January 06, 2018, 11:46:06 AM
I’ll search for it,thanks.. I have managed to define around 35 maps on my bin manually and using friends’ help. How do I define an XML? Is there a manual? The pattern, where do i get it from? Are those values like contained always in a map and that’s how it identifies them? I checked some of the patterns but they are not incremental rather than mixed values..

Thanks!
It’s all explained by prj in the first few posts.


Title: Re: ASM based map locator platform
Post by: antaljani on March 15, 2018, 01:23:31 PM
Hello Everybody,

When I try to compile this java stuff, I getting an error message:
Code:
Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on project maplocator-ui: Compilation failure
[ERROR] /Users/antaljanos/Documents/GitHub/me7-tools/maplocator-ui/src/main/java/com/prj/tuning/maplocator/export/xdf/model/Project.java:[17,8] com.prj.tuning.maplocator.export.xdf.model.Project is not abstract and does not override abstract method getConstants() in com.prj.tuning.xdf.binding.XdfProject

I cannot find any solution in this thread...
Is there anybody, who can help me?

I'm using Maven latest version and also Java SDK latest versioin.

Btw, why the compiled jar file does not exist in github?

Thanks in advance.


Title: Re: ASM based map locator platform
Post by: marcjero on June 23, 2018, 05:38:07 AM
Hello,

thank you a million for sharing this nice tool. I would like to extend it in order to support other Bosch ecus (MEG1.x). I have several question before starting.

-XML Plugin seems to have hardcoded support for Audi and Volvo ecus. Should I patch the XML Plugin code or write my own Java plugin ? Maybe it's possible to make the XML Plugin more generic ?

-The ECU I'm working on includes several variants of the same maps (for different engine power) Problem is the code calling the maps is quite similar also, so it's very difficult to find a single match pattern. What is happening when there are several matching candidates ? Is the first match returned ?

BTW I just found this tiny disassembler for C166 : https://github.com/hn/c166-dis
Works pretty well and is enough for me as I can't afford the IDA pro license.


Title: Re: ASM based map locator platform
Post by: marcjero on June 24, 2018, 05:49:04 AM
Hi,

I did a patch in order to support meg ecu. Fork is available here : https://github.com/jmarcgit/me7-tools (https://github.com/jmarcgit/me7-tools)
 I will do a pull request if you agree.

Anyway I'm struggling when defining some matchers. In example below I'm trying to catch 0x03AA address. Problem is there are very similar code around. There are pages of code like this. I don't think that relying on position will be reliable.

Code:
F6F5ECF9  MOV      0x0000F9EC,R5
00023216  E6F1BE03  MOV      R1,#0x03BE
0002321A  C2F41DF7  MOVBZ    RL2,0x0000F71D
0002321E  C2F5EAF9  MOVBZ    RL2,0x0000F9EA
00023222  DA060456  CALLS    0x06'0x5604
00023226  F6F5EAF9  MOV      0x0000F9EA,R5
0002322A  E6F1AA03  MOV      R1,#0x03AA
0002322E  C2F41DF7  MOVBZ    RL2,0x0000F71D
00023232  C2F5F0F9  MOVBZ    RL2,0x0000F9F0
00023236  DA060456  CALLS    0x06'0x5604
0002323A  F6F5F0F9  MOV      0x0000F9F0,R5
0002323E  E6F19203  MOV      R1,#0x0392
00023242  C2F41DF7  MOVBZ    RL2,0x0000F71D
00023246  C2F5F2F9  MOVBZ    RL2,0x0000F9F2
0002324A  DA060456  CALLS    0x06'0x5604
0002324E  F6F5F2F9  MOV      0x0000F9F2,R5
00023252  E6F18603  MOV      R1,#0x0386
00023256  C2F41DF7  MOVBZ    RL2,0x0000F71D
0002325A  C2F5F4F9  MOVBZ    RL2,0x0000F9F4
0002325E  DA060456  CALLS    0x06'0x5604


Title: Re: ASM based map locator platform
Post by: prj on June 24, 2018, 03:11:11 PM
This is abandonware for me at this point, as I just don't have time :)
Forking is alright, I should give someone access to the repo who wants to merge stuff and so on, if anyone is interested.


Title: Re: ASM based map locator platform
Post by: marcjero on June 24, 2018, 04:41:50 PM
I actually forked the nyet's fork. So I will pull my changes to him. I think nyet is still active on the project.

I know how time consuming this kind of work is. I appreciate the quality of your work.

I get great results (3 maps are already working) but I'm struggling with some maps (axes in fact) because I can't find unique matching patterns for them. I could implement a new search logic but I don't know how to solve this problem for now.


Title: Re: ASM based map locator platform
Post by: marcjero on June 25, 2018, 04:23:01 AM
I think I found a solution for my use case. Just look to the code below.
Code:
00054160  F6F5E682   MOV      0x000082E6,R5
00054164  F2F88087   MOV      R8,0x00008780
00054168  F6F8E882   MOV      0x000082E8,R8
0005416C  8A380170   JB       CC24.7,0x00054172
00054170  0D03       JMPR     CC_UC,0x00054178
00054172  E6F1FE0A   MOV      R1,#0x0AFE
00054176  0D02       JMPR     CC_UC,0x0005417C
00054178  E6F17E0A   MOV      R1,#0x0A7E
0005417C  F2F5E882   MOV      R5,0x000082E8
00054180  F2F43C87   MOV      R4,0x0000873C
00054184  E087       MOV      R7,#8
00054186  DA078059   CALLS    0x07'0x5980


00022DE6  F6F54E87   MOV      0x0000874E,R5
00022DEA  E6F18000   MOV      R1,#0x0080
00022DEE  C2F415F7   MOVBZ    RL2,0x0000F715
00022DF2  C2F53C87   MOVBZ    RL2,0x0000873C
00022DF6  DA07A658   CALLS    0x07'0x58A6

First block is the area processing a map. Second block is processing an axis used by the map. Map address is 0x0A7E and axis address is 0x0080. We see that the address 0x0000873C is used in both code sections.

So in order to identitfy my axis address I need first to lookup the table section, get the shared address (0x0000873C) and then process a lookup to the axis section using the shared address. I think this was discussed earlier to this discussion.

Looks very exciting and quite easy to implement.

Well I did it :

Code:
E6 F1 XX XX 0D 02 E6 F1 XX XX F2 F5 XX XX F2 F4 MMXX XX E0 87 DA XX XX XX F7 FA XX XX 8A XX XX XX 8A XX XX XX 8A|F6 F5 XX XX E6 F1 MMXX XX C2 F4 XX XX C2 F5 $$0 DA

This expression is just doing a lookup to the 0x0080 address of the above example. You can pipe as many expressions as you need. Each match can be reused in the following expressions using the $$n notation. For instance $$0 is inserting the address returned by the first expression evaluation result. I think this can support complex scenario. You can use several $$ notations in a single expression. Feedbacks are welcome. Github updated so you can test now with your own definitions.



Title: Re: ASM based map locator platform
Post by: nyet on June 26, 2018, 02:20:43 PM
I actually forked the nyet's fork. So I will pull my changes to him. I think nyet is still active on the project.

I know how time consuming this kind of work is. I appreciate the quality of your work.

I get great results (3 maps are already working) but I'm struggling with some maps (axes in fact) because I can't find unique matching patterns for them. I could implement a new search logic but I don't know how to solve this problem for now.

I'm willing to maintain a canonical repo. Thanks for your PR!

https://github.com/nyetwurk/me7-tools/


Title: Re: ASM based map locator platform
Post by: marcjero on June 26, 2018, 02:37:56 PM
You are welcome. This tool is amazing. And I can now lookup almost anything I need. I just have to learn the meaning of the tables. Smart MEG is using MAP when most ME ecus use MAF and german translation is not always great. ::)

I got an idea about another useful feature to add. When you have 2 similar bins with different xdfs definitions (addresses are different) it's pain to compare the tables using Tunerpro. I think it could be useful to automatically move the data from a bin to another bin when the identified addresses are different. Once done comparing or copying data in Tunerpro would be very easy because a single xdf will be required.


Title: Re: ASM based map locator platform
Post by: prj on June 26, 2018, 02:41:50 PM
Use WinOLS and that problem won't be there :)
For comparing you can use the demo and newer versions of OLS have xdf import.


Title: Re: ASM based map locator platform
Post by: marcjero on June 26, 2018, 03:35:55 PM
nice to know.

btw I have an idea about the design:
-define a second optional file field (call the first one master bin file)
-so when analysis is run : the 2 files can be processed at the same time and results are displayed using 2 columns
-when xdf export is run : it only applies to the master file
-when 2 files are selected, a "migrate data" button is enabled. When clicking to this button : a copy of the master file is done and then data from the second file are copied into the copy of the master file (simple address translation affair)

It's just a general design idea. Does it make sense for non winols users ?







Title: Re: ASM based map locator platform
Post by: nyet on June 26, 2018, 04:06:09 PM
Does it make sense for non winols users ?

Just about everything OLS does that TunerPro doesn't would be useful, IMO. Most would say "just use OLS" but it would be nice if more things could be done without it.


Title: Re: ASM based map locator platform
Post by: marcjero on June 26, 2018, 11:45:54 PM
Just about everything OLS does that TunerPro doesn't would be useful, IMO. Most would say "just use OLS" but it would be nice if more things could be done without it.

Yes winols cost is about 1000$ and tunerpro is free, so it's not surprising to learn that winols is vastly superior.
 What do you think about my design proposal ?


Title: Re: ASM based map locator platform
Post by: nyet on June 27, 2018, 12:44:34 AM
Yes winols cost is about 1000$ and tunerpro is free, so it's not surprising to learn that winols is vastly superior.
 What do you think about my design proposal ?

Yes, but I think it should be a separate tool.


Title: Re: ASM based map locator platform
Post by: marcjero on June 27, 2018, 02:29:45 AM
Yes it can be a different tool. It can be based on XDF files. I think it will be better then is to use the XML ids as XDF tables uniqueIds.(instead of addresses)
A single tool would make things simpler for users but will be harder to maintain.


Title: Re: ASM based map locator platform
Post by: marcjero on July 01, 2018, 01:25:23 PM
Hi,

I'm very confused about the content of the MEG 1.1 ECU. It looks like that most tables and axis are duplicated and available in 2 areas : The first one starts at 0x10000 and the second one starts at 0x18000
For instance the KZFW table offset is 0x2150. I was thinking  the address was 12150 but it happens that 1A150 is working too. Both adresses are containing the same data.

This happens for MEG1.1 bin files only. For MEG1.0 the 0x18000 region is empty. Do you know why these data are duplicated ?



Title: Re: ASM based map locator platform
Post by: prj on July 01, 2018, 01:53:49 PM
Because same ECU is used for 2 different cars, for example auto/manual and instead of only switching a few maps the entire cal area is duplicated, and the switch is done using DPP.


Title: Re: ASM based map locator platform
Post by: marcjero on July 01, 2018, 03:18:35 PM
I guess then the 0x18000 are is for the Roadster and the 0x10000 area is for the Fortwo. Thanks.


Title: Re: ASM based map locator platform
Post by: 360trev on September 08, 2018, 06:31:13 AM
I know prj has done all of this great research and work 6+ years ago now I still think a fresh pair of eyes by another time served developer (me) could potentially (using some different approaches) move the game on further too.

I think you may actually be able to develop a ME7.x heuristic search model which could identify 100% of all maps in any ME7.x firmware almost automatically without lots of community contributed needle/masks. Infact I am currently working on such an approach...

How?
By concentrating on and identifying all (only tens of) indirect Lookup functions in ME7.x firmwares (and dumped IROM!) which all table accesses seem to go through anyway (!) at least all of the ones I've manually explored so far,  you can then find those references 'generically' in the entire machine code of the functions which utilize table lookups.

Q. Why do this? Well once you know what function your dealing with you know the format of the table (sizes of the fields, etc.) so its easy to dump the tables correctly. No guessing!

Then you can walk through the firmware image and discover all references to the Lookup functions and thus ALL tables (every single one). The various lookup's do things slightly differently depending on the number of its or rows/columns in any given table and this makes it possible to write a generic search function which catches all tables...

So by identifying the lookup's first and then working backwards from those known 'hits' I think you can infact locate all of the tables that the rom uses and correctly extract the row/columns directly out of the discovered code (as well as extract the segmentation information to discover the table itself, again automatically).

I'd be interested in prj's thoughts on this...


Title: Re: ASM based map locator platform
Post by: nyet on September 08, 2018, 04:57:59 PM
I absolutely agree that minimizing per ECU needle/mask requirement is a good idea, and finding more generic ways of locating both ram variables and map locations would be of great utility.


Title: Re: ASM based map locator platform
Post by: woj on September 09, 2018, 12:47:08 AM
I have some thoughts, because I actually more or less went all the way through this, using the same principle idea, but in a slightly different way. My very own private ST10 disassembler does that. Goes through the code once, finds all calls to scale reading routines, identifies scale index variables and the corresponding scale sizes, then goes through the code again and finds all maps. For scale integrated maps does the same thing, only in one pass. And yes, it does have an almost 100% hit rate. It even identifies double indirect referenced maps (very common in some of my bins, when m/a gearbox map is selected through an additional pointer reference). All is needed is to identify the address to the IROM block where the OS routines for map reading are. So far this part is manual, but can be easily automated too (I work with only one family of ECUs, so I don't really know how different it is on the other MEs).

You can easily as well do it as an IDA script, I have chosen a different route because of he language I use for this, that has pattern recognition capabilities beyond your imagination.

Now, my actual points. This method is not 100% bullet proof, you'd still get weird code, like an indexing variable held in a register for couple of tens of instructions, with disassembly you can do something about that, with binary code matching not so much, unless you in fact turn it into a disassembler.

The second point is this: so what? (Don't read this as ball-breaking, but encouragement ;)) I mean finding map boundaries is one problem and not a difficult one, finding what these maps actually are and make ME version independent is another thing. For my ECU I am practically interested in knowing around 400 maps (or single parameters / bits!) and sorting which one is which (eg. which KFPED is which, sport, reverse, automatic, manual gearbox, etc.). This I have not found a solution to yet, I mean one that would have considerable automation. This is what PRJ did in his map locator, but it requires manually crafted patterns and is not automatic in this respect (correct me if I am wrong). If I understand it correctly, it works well if you have two ECUs of the same family, you identify maps in one and created a pattern, then it will get you quickly to these maps in another ECU based on the same ME version. (No PRJ, I have not used your program, only looked through it once, so excuse me if I got it all wrong).


Title: Re: ASM based map locator platform
Post by: 360trev on September 09, 2018, 08:02:47 AM
I have some thoughts, because I actually more or less went all the way through this, using the same principle idea, but in a slightly different way. My very own private ST10 disassembler does that. Goes through the code once, finds all calls to scale reading routines, identifies scale index variables and the corresponding scale sizes, then goes through the code again and finds all maps. For scale integrated maps does the same thing, only in one pass. And yes, it does have an almost 100% hit rate. It even identifies double indirect referenced maps (very common in some of my bins, when m/a gearbox map is selected through an additional pointer reference). All is needed is to identify the address to the IROM block where the OS routines for map reading are. So far this part is manual, but can be easily automated too (I work with only one family of ECUs, so I don't really know how different it is on the other MEs).

You can easily as well do it as an IDA script, I have chosen a different route because of he language I use for this, that has pattern recognition capabilities beyond your imagination.

Now, my actual points. This method is not 100% bullet proof, you'd still get weird code, like an indexing variable held in a register for couple of tens of instructions, with disassembly you can do something about that, with binary code matching not so much, unless you in fact turn it into a disassembler.

The second point is this: so what? (Don't read this as ball-breaking, but encouragement ;)) I mean finding map boundaries is one problem and not a difficult one, finding what these maps actually are and make ME version independent is another thing. For my ECU I am practically interested in knowing around 400 maps (or single parameters / bits!) and sorting which one is which (eg. which KFPED is which, sport, reverse, automatic, manual gearbox, etc.). This I have not found a solution to yet, I mean one that would have considerable automation. This is what PRJ did in his map locator, but it requires manually crafted patterns and is not automatic in this respect (correct me if I am wrong). If I understand it correctly, it works well if you have two ECUs of the same family, you identify maps in one and created a pattern, then it will get you quickly to these maps in another ECU based on the same ME version. (No PRJ, I have not used your program, only looked through it once, so excuse me if I got it all wrong).

Thank you woj for an intelligent well thought out reply.

I have actually just been writing a C167 dissassembler (as part of this just to become more familiar with the machine code, syntax and operations you can do with the various instructions) which I will also host into my github account shortly. I needed this to be able to 'walk' through the machine code sequences and find if they are 2 or 4 byte lengths as part of my scanning routines. IDA has a loverly abstracted interface to accomplish this via a plugin or IDA scripts (which I used initially to prove some concepts) but I want my work to be non incumbered by any commerical licensing requirements at all hence doing my own thing. As such I'm not interested in the slightest in using IDA scripting since in my (humble) opinion its also very useful to be able to do all of this work without utilizing ANY 3rd party tools which are commerical, no IDA, no WinOLS, etc. It should ideally be fully self contained with all the source-code available for inspection by the reader. Also makes it less likely to be broken by any updates ;)

Its all well and good keeping all this stuff private (and I absolutely respect your rights to do this) but in my situation where I am not making a living from any of this, its absolutely just a hobby I will open source all of my work as a help to others and so it can be improved on in the future or ported to other platforms. I'm not a huge fan of Java (hate it to be honest despite having had to use it before in some larger scale industry projects in a former life!) I just didn't want to use prj's map finder as I felt I wanted to take it in a different direction anyway.

In terms of the so what points I have a clever way (which for now I will keep under wrap [need to have some surprises!) which I automatically identify the map names from. I've done extensive work on my own rom now and as such I've identified over 3,000 variables and functions (!) as well as documented the all the xref's back from variables to defining functions. This is a very powerful database which I hope to deploy, I've already done some preliminary work on this and optimistically I think I can get the majority of functions and variables in any ME7.x rom defined automatically.
 
I'll keep you all posted as I progress the concept...


Title: Re: ASM based map locator platform
Post by: 360trev on September 09, 2018, 08:06:49 AM
I posted this before here...

Code:
atic void PlugIn_process(int iArg)
{
  char funcName[MAXSTR];
  char filename[MAXSTR];
  char mnem[MAXSTR];
  char tmp[1024*512];
  int offset=0;
  int inst_count,func_len;
  unsigned int x, num_funcs, num_segments;
  ea_t addr, start_addr;
  asize_t our_size;
  flags_t flags;
  func_t *f=0;
  segment_t *seg=0;

  // We are only interested in segments containing code.
  seg = getnseg(0);
  num_funcs = get_func_qty();

         // Loop through each function
  msg("Found <%d> functions\n",num_funcs);

for (x = 0; x < num_funcs; x++)
        {
        f = getn_func(x);      // get ptr to the function itself
if(f == 0) {
msg("func not found at address %p\n");
}
else
{
// start address of this function..
start_addr = f->startEA;
addr = start_addr;

get_func_name(addr, funcName, sizeof(funcName)-1); // find the function name
//      set_name(addr, new_funcName, SN_NOWARN);

flags = get_flags_novalue(addr);
msg("Found %-32.32s: at %p, func_flags(%lx)",funcName,addr,flags);
//  Add name if there's no meaningful name assigned.
if(has_name(flags) != 0)       { msg("(customized name   ) "); }
if(has_dummy_name(flags) != 0) { msg("(autogenerated name) "); }
if(has_auto_name(flags) != 0)  { msg("(has_auto_name     ) "); }

//
// calc number of instructions
//
inst_count = 0;
addr = start_addr; // start of function
func_len   = ((f->endEA)-(start_addr));

// count the number of instructions within the function
for (; addr < f->endEA;) {
  // Get the flags for this address
  flags = get_flags_novalue(addr);
  // Only look at the address if it's a head byte, i.e. the start of an instruction and is code.
  if (isHead(flags) && isCode(flags))  {   
  inst_count++; // increase number of instructions counter we have seenn within this function
  }
  our_size = get_item_size(addr); // get number of bytes within this function
  addr += our_size;                 // move addres pc to next instruction...
}
//
// at this point 'inst_count' is total number of asm instructions within the buffer
//
msg(" func len = %-8d bytes (%-8d instructions)\n",func_len, inst_count );

sprintf(filename,"c:\\bin\\%p_%s_%d.bin",start_addr,funcName,inst_count);

#if 1
addr = start_addr; // start of function
offset=0;
// lets now loop through the instructions in each function
for (; addr < f->endEA;)
{
  // Get the flags for this address
  flags = get_flags_novalue(addr);

    // get the size of this item (e.g. instruction length)
  our_size = get_item_size(addr);

  // Only look at the address if it's a head byte, i.e.
  // the start of an instruction and is code.
  if (isHead(flags) && isCode(flags))
  {   
char instruction[16];

// Fill the cmd structure with the disassembly of the current address and get the mnemonic text.
ua_mnem(addr, mnem, sizeof(mnem)-1);

ua_ana0(addr);
ua_mnem(addr, instruction, sizeof(instruction));
tag_remove(instruction, instruction, sizeof(instruction));

// lets view the mnemonic of *this* address
// msg("%p:[%-2.2d] ", addr, our_size);
#if 1
msg(".");
get_many_bytes(addr,&tmp[offset],our_size);
offset += our_size;
#else
for(int j=0;j < our_size; j++)
{
tmp[j] = get_byte(addr+j);
}
#endif
// show the hex dump of the instruction
// hex_dump((unsigned char *)&tmp[offset], our_size);
// show the mnemonic of the instruction
// msg(" %-8.8s\n",instruction);
  }
  // move addres pc to next instruction...
  addr += our_size;
}

// dump it..
msg("dumping %s\n",filename);
save_file(filename, (unsigned char *)tmp, (size_t)offset);

#endif
}//end if

  }//end for
}

Using IDA's instruction walking interface this alone dumps every function in a ME7... I'm sure its easy to do with IDA scripting. If you just add the masking out the reloc's you have a very powerful way to cross reference functions across different roms with very little work..
 


Title: Re: ASM based map locator platform
Post by: woj on September 09, 2018, 09:07:57 AM
(PRJ is going to be thrilled to see all this ranting in his thread :D)

Your motivation for not using IDA and such is more or less exactly as mine. In fact, the only commercial tool I use (and that is not even really commercial, just semi-closed source) is TunerPro, all of the other software I use is mine, including a flasher and a macro assembler to patch code.

These are poor excuses to keep my software private, but there are some: 1. it is very targeted in its current form, so using it in a more general context than my ECU won't work. 2. Releasing things means it needs to have decent SE quality, my stuff does not and I don't have time to fix it. (It's so bad, that I cannot even trace my own way of thinking in software I have written half a year before). 3. Releasing things usually causes a shit-storm of stupid questions and I simply cannot deal with it. 4. Some software requires custom made hardware, and without it it's useless.

So, what I do instead is I release bit and independent pieces of information here and there so that the inclined ones can make some use of it.

I am curious about your approach, but reading "database" scares me a bit.

A possible future direction to consider is to go beyond ST10 / ME, and develop something for TriCore for example ;) ;)



Title: Re: ASM based map locator platform
Post by: prj on September 09, 2018, 01:51:32 PM
I mean rant on, why not ;)

I think ME7 is ... well sorta dead for this stuff.
As in most stuff that exists is already decently defined or there is a hex/a2l for a similar firmware that can be found...


Title: Re: ASM based map locator platform
Post by: woj on September 09, 2018, 02:07:14 PM
Well, that's one opinion and I partly agree with it. If one's a pro then defo the new stuff counts much more. But then, you get tons of new people getting on here every day crying for help with maps, LC/NLS patches, gargles/pops, and what not. As long as there will be cars with ME7 there will be non zero interest. From amateurs. (Myself, I seem to be always two generations of ECUs behind the mainstream).

Having said that, I'd really like to see cutting edge tools for newer architectures. I recently realised Fiat has swapped my beloved ME7.9.10 technology for my beloved Fiat t-jet engine with ME17.3.0 which is a whole new ball game (for me at least). One that I would eventually want to get into, just some fucker some time ago decided a day should have only 24 hours :/


Title: Re: ASM based map locator platform
Post by: eliotroyano on September 10, 2018, 06:36:00 PM
Maybe this kind of MAP locator tool can be tweaked for newer software and platforms. Of course if are not totaly encrypted or masked.


Title: Re: ASM based map locator platform
Post by: nyet on September 12, 2018, 10:05:11 PM
Also: detect ram locations to extend/replace ME7Info


Title: Re: ASM based map locator platform
Post by: 360trev on September 21, 2018, 03:48:44 AM
Also: detect ram locations to extend/replace ME7Info

Yes, absolutely agree on this one as it finds virtually nothing for a Ferrari rom and I want to do realtime telemetry for my track days so this is a must..

I've just been finishing my C16x/ST10 dissassembler which I will spin off into its own seperate project on github as a shared library and simple frontend for other tools to use within them. This is already working in basic format...

Code:
-[ Basic Firmware information ]-----------------------------------------------------------------

>>> Scanning for ROM String Table Byte Sequence #1 [info]

found needle at offset=0x2cc04
found table at offset=00019A5C.

         0X2CC04 [+0   ]: mov      var_b, Rw_a                          [4] F6 F4 42 E2
         0X2CC08 [+4   ]: mov      var_b, Rw_a                          [4] F6 F5 44 E2
         0X2CC0C [+8   ]: jnb      cbitaddr_a, cbitaddr_c, Rel_b        [4] 9A 22 05 10
         0X2CC10 [+12  ]: movb     Rb_a,   #cdata16_b                   [4] E7 F8 11 00
         0X2CC14 [+16  ]: movb     var_b, Rb_a                          [4] F7 F8 0A E2
         0X2CC18 [+20  ]: jmpr     ccc_0000,Rel_l_a                     [2] 0D 04
         0X2CC1A [+22  ]: movb     Rb_a,   #cdata16_b                   [4] E7 F8 14 00
         0X2CC1E [+26  ]: movb     var_b, Rb_a                          [4] F7 F8 0A E2
         0X2CC22 [+30  ]: mov      Rw_a,   #cdata16_b                   [4] E6 F4 C8 CD
         0X2CC26 [+34  ]: mov      Rw_a,   #cdata16_b                   [4] E6 F5 82 00
         0X2CC2A [+38  ]: mov      var_b, Rw_a                          [4] F6 F4 32 E2
         0X2CC2E [+42  ]: mov      var_b, Rw_a                          [4] F6 F5 34 E2
         0X2CC32 [+46  ]: rets                                          [2] DB 00
         0X2CC34 [+48  ]: mov      Rw_b,   Rw_a                         [2] 88 60
         0X2CC36 [+50  ]: mov      Rw_a,   #cdata16_b                   [4] E6 F4 86 2B
         0X2CC3A [+54  ]: mov      Rw_a,   #cdata16_b                   [4] E6 F5 00 00
         0X2CC3E [+58  ]: mov      var_b, Rw_a                          [4] F6 F4 B2 E1
***
Idx=1   { 182542.000              } 0x101a1 : VMECUHN [Vehicle Manufacturer ECU Hardware Number SKU]
Idx=2   { 0261204841              } 0x1018b : SSECUHN [Bosch Hardware Number]
Idx=4   { 0000000000              } 0x10196 : SSECUSN [Bosch Serial Number]
Idx=6   { F131 CHALLENGE          } 0x10177 : EROTAN  [Model Description]
Idx=8   { R.BOSCH001              } 0x19a50 : TESTID
Idx=10  { 069117/15H52CE2         } 0x10167 : DIF
Idx=11  { 0691175H                } 0x1015e : BRIF

>>> Scanning for EPK information [info]

found needle at offset=0x27902.
EPK: @ 0x10029 { /1/ME7.3/69/117/F131_US//15h52ce2/151299/ }

-[ PUKANS Air Pulsation correction Table ]-----------------------------------------------

Function byte-sequence found @ 0x4b8c2, needle offset is +44 bytes, seg offset is +48 bytes

         0X4B8C2 [+0   ]: mov      Rw_a,   #cdata16_b                   [4] E6 FC E4 0A
         0X4B8C6 [+4   ]: movbz    Rb_a,   cmm_b                        [4] C2 FD 6C F8
         0X4B8CA [+8   ]: movbz    Rb_a,   cmm_b                        [4] C2 FE 7B F9
         0X4B8CE [+12  ]: calls    cseg_a, ccadr_b                      [4] DA 00 12 73
         0X4B8D2 [+16  ]: movb     var_b, Rb_a                          [4] F7 F8 51 8B
         0X4B8D6 [+20  ]: movbz    Rb_a,   cmm_b                        [4] C2 F4 52 8B
         0X4B8DA [+24  ]: movbz    Rb_a,   cmm_b                        [4] C2 F5 51 8B
         0X4B8DE [+28  ]: mul      Rw_a,   Rw_b                         [2] 0B 45
         0X4B8E0 [+30  ]: mov      var_b, Rw_a                          [4] F6 07 CE 9D
         0X4B8E4 [+34  ]: mov      Rw_a,  cmm_b                         [4] F2 F4 0E FE
         0X4B8E8 [+38  ]: mov      Rw_a,   [Rw+]                        [2] 98 90
         0X4B8EA [+40  ]: rets                                          [2] DB 00
         0X4B8EC [+42  ]: mov      Rw_a,   #cdata16_b                   [4] E6 FC AA 05
         0X4B8F0 [+46  ]: mov      Rw_a,   #cdata16_b                   [4] E6 FD 06 02
         0X4B8F4 [+50  ]: movbz    Rb_a,   cmm_b                        [4] C2 FE B8 8B
         0X4B8F8 [+54  ]: calls    cseg_a, ccadr_b                      [4] DA 82 78 97
         0X4B8FC [+58  ]: movb     var_b, Rb_a                          [4] F7 F8 D0 9D
         0X4B900 [+62  ]: rets                                          [2] DB 00
         0X4B902 [+64  ]: mov      Rw_a,  cmm_b                         [4] F2 F4 2C 8F
         0X4B906 [+68  ]: cmp      ...                                  [2] 48 40
***

PUKANS
    Long identifier:           Pulsation correction dependent on intake air temperature.
    Display identifier:
    Address:                   0x8185aa
    Value:

 No.           |        0        1        2        3        4        5        6        7
            PHY|    11.00    37.00    64.00    91.00   104.00   117.00   131.00   171.00
 --------------+------------------------------------------------------------------------
            PHY|   1.0859   1.0391   1.0000   0.9688   0.9453   0.9375   0.9219   0.8828


    Cells:
      Unit:
      Conversion name:
      Conversion formula:      f(phys) = 0.0 + 0.007812 * phys
      Data type:               UBYTE
    X-axis:
      Unit:                    Grad C
      Conversion name:
      Conversion formula:      f(phys) = 0.0 + 1.000000 * phys
      Data type:               UBYTE

And I'm just finishing the bits required to make it into a proper disassembler and single line assembler.

The idea is you can then upload patched code at runtime (during telem sessions) as well as realtime map editing. If you have a function uplad feature you can add features on the fly without having to reflash which for my usage is a must as I want to be able to rapidly develop and test without the whole reflash cycle which is just too time consuming...

I needed the dissassembler so I could easily work on runtime/offline patching...

Watch this space a big update is coming soon on my ME7 Swiss Army Knife tool ;)



Title: Re: ASM based map locator platform
Post by: 360trev on September 21, 2018, 03:52:46 AM
Maybe this kind of MAP locator tool can be tweaked for newer software and platforms. Of course if are not totaly encrypted or masked.

Yes, I've already acquired a Hitex Shield Buddy which boasts a fully programmable Infineon TriCore TC275 with full SDK and compilers, etc. I will get to this in good time.. ;)


Title: Re: ASM based map locator platform
Post by: taco on December 27, 2018, 07:35:37 AM
The tool is amazing. I used nyet's fork and it found a nice shortlist to add to my "in progress" def.
ME7Logger can monitor many different items....
If the source is available maybe something from it can be adapted to help the map finder?


Title: Re: ASM based map locator platform
Post by: zarboz on March 11, 2019, 09:59:10 AM
I am having a slight issue with the software where the [Me7XmlPlugin] will print the proper hex offset but the xdf / UI does not display the proper hex address

It is either how the app is temporarily storing the offset when identifying the addr or its the string.format formatting an existing real hex offset into a null hex offset


Terminal shows:
[Me7XmlPlugin] Pattern for CLRHK found at: 0x782E8

XDF export shows
<title>CLRHK</title>
        <XDFAXIS id="z">
            <decimalpl>2</decimalpl>
            <EMBEDDEDDATA mmedtypeflags="0x0" mmedaddress="0x27F2" mmedelementsizebits="8"/>
            <MATH equation="0.000000+X*1.000000">
                <VAR>
                    <id>X</id>
                </VAR>
            </MATH>
        </XDFAXIS>

and UI also shows 27F2

but real offset is : 0x782E8 as reported by the initial Me7XmlPlugin which oddly enough uses the same string.format command

if anyone can help me out it would be awesome
TLDR:
Me7XmlPlugin reports proper offset in terminal but UI / XDF show completely different offset

my branch is here:
https://github.com/zarboz/me7-tools




Title: Re: ASM based map locator platform
Post by: marcjero on July 19, 2022, 05:59:13 PM
Hi,

I have some trouble implementing static axes with the tool. My TVUB axis is calculated so it's not possible to manage it as a table. On top of that the axis definition is located just before the TVUB table and code is referencing the axis definition. The axis defintion is 6 bytes long, so basically I have to add a 6 bytes offset to the final address in order to locate the TVUB.

That's an idea I got to solve that:
1/ allow to define static content for maps, so we could define static axes this way
2/ Define the TVUB_AXIS with static content and then declare TVUB as following the TVUB axis

I don't know if Tunerpro allows to define tables with data stored in the XDF file (why would it do that really?) But one can define axis values in the XDF file for sure.
Therefore the XDF export could filter out table defintions that contains data and populate axis values instead of using axis references.

What do you think about this proposal ?

here is the A2L defintion of the TVUB axis:
Code:
    /begin CHARACTERISTIC
      TVUB "EV-Spannungskorrektur"
      CURVE 0x1466C DAMOS_FKL1 0 UTS 0.000000 10.240000
      /begin AXIS_DESCR
        FIX_AXIS UBATT UBQ 17 0.000000 26.037190
        FIX_AXIS_PAR 0 4 17
        EXTENDED_LIMITS 0.000000 26.037200
      /end AXIS_DESCR






Title: Re: ASM based map locator platform
Post by: marcjero on August 14, 2022, 01:40:15 PM
Hi,

For information I did major changes is the software. My repo is here : https://github.com/jmarcgit/me7-tools

Now it's compatible with latest JDK. (11-18 should be ok). Works on some ARM64 devices including Apple M1.
Updated the old libraries.

Calculated axis are implemented.

I think that doing pull requests doesn't make sense now. Too many changes.
I'm ok to maintain this code.


Title: Re: ASM based map locator platform
Post by: nyet on August 14, 2022, 08:46:55 PM
Good call; I don't have the time to maintain my branch, apologies.

Thank you.


Title: Re: ASM based map locator platform
Post by: marcjero on August 15, 2022, 05:20:12 AM

I found the cascading match patterns really useful because it makes possible to link tables to the axes it really uses automatically. I do the link using a memory address in which an axis reference s stored. In my case (Smart MEG1.x) I observed few different rpm axes (values and length can vary). And the same table can use one or another axis from a bin to another.

Code:
<?xml version="1.0" encoding="UTF-8"?>
<map xmlns="http://prj-tuning.com/mapdef" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://prj-tuning.com/mapdef mapdef.xsd ">
<id>NIST@KFLDS</id>
<title>NIST@KFLDS - RPM Breakpoints</title>
<pattern>E6 F1 XX XX 0D 02 E6 F1 XX XX F2 F5 XX XX F2 F4 MMXX XX E0 87 DA XX XX XX F7 FA XX XX 8A XX XX XX 8A XX XX XX 8A|F6 F5 XX XX E6 F1 MMXX XX C2 F4 XX XX C2 F5 $$0 DA</pattern>
<conversion>
<factor>25</factor>
<signed>false</signed>
</conversion>
</map>

This is an example of dynamic RPM axis lookup (for KFLDS table)

First part of the pattern (before pipe) is locating the KFLDS table. The address after F2 F4 is the address the first axis reference.

This address is captured and reused in second pattern after C2 F5 and then the axis final address can be found just after E6 F1.

Looks like a bit complicated but it's not. The first pattern is a copy paste of the table pattern except that the MM mark is set at a different place.
It increases the accuracy of the axis lookup which is done in the second pattern.

I never tried anything more complicated but any number of patterns can be defined and matching values can be referenced using $$0, $$1,... $$n style expressions
Basically this feature is usefull when you have to process some indirect addressing.

From my experience the ecu code is built using the same coding patterns extensively.
This means that once an axis pattern like this one is discovered it can be reused a lot.