360trev
Full Member
Karma: +68/-2
 Offline
Posts: 235
|
 |
« Reply #135 on: September 08, 2018, 06:31:13 AM »
|
|
|
I know prj has done all of this great research and work 6+ years ago now I still think a fresh pair of eyes by another time served developer (me) could potentially (using some different approaches) move the game on further too.
I think you may actually be able to develop a ME7.x heuristic search model which could identify 100% of all maps in any ME7.x firmware almost automatically without lots of community contributed needle/masks. Infact I am currently working on such an approach...
How?
By concentrating on and identifying all (only tens of) indirect Lookup functions in ME7.x firmwares (and dumped IROM!) which all table accesses seem to go through anyway (!) at least all of the ones I've manually explored so far, you can then find those references 'generically' in the entire machine code of the functions which utilize table lookups.
Q. Why do this? Well once you know what function your dealing with you know the format of the table (sizes of the fields, etc.) so its easy to dump the tables correctly. No guessing!
Then you can walk through the firmware image and discover all references to the Lookup functions and thus ALL tables (every single one). The various lookup's do things slightly differently depending on the number of its or rows/columns in any given table and this makes it possible to write a generic search function which catches all tables...
So by identifying the lookup's first and then working backwards from those known 'hits' I think you can infact locate all of the tables that the rom uses and correctly extract the row/columns directly out of the discovered code (as well as extract the segmentation information to discover the table itself, again automatically).
I'd be interested in prj's thoughts on this...
|
|
|
|
|
Logged
|
|
|
|
|
nyet
|
|
« Reply #136 on: September 08, 2018, 04:57:59 PM »
|
|
|
I absolutely agree that minimizing per ECU needle/mask requirement is a good idea, and finding more generic ways of locating both ram variables and map locations would be of great utility.
|
|
|
|
|
Logged
|
ME7.1 tuning guideECUx PlotME7Sum checksumTrim heatmap toolPlease do not ask me for tunes. I'm here to help people make their own. Do not PM me technical questions! Please, ask all questions on the forums! Doing so will ensure the next person with the same issue gets the opportunity to learn from your ex
|
|
|
|
woj
|
|
« Reply #137 on: September 09, 2018, 12:47:08 AM »
|
|
|
I have some thoughts, because I actually more or less went all the way through this, using the same principle idea, but in a slightly different way. My very own private ST10 disassembler does that. Goes through the code once, finds all calls to scale reading routines, identifies scale index variables and the corresponding scale sizes, then goes through the code again and finds all maps. For scale integrated maps does the same thing, only in one pass. And yes, it does have an almost 100% hit rate. It even identifies double indirect referenced maps (very common in some of my bins, when m/a gearbox map is selected through an additional pointer reference). All is needed is to identify the address to the IROM block where the OS routines for map reading are. So far this part is manual, but can be easily automated too (I work with only one family of ECUs, so I don't really know how different it is on the other MEs). You can easily as well do it as an IDA script, I have chosen a different route because of he language I use for this, that has pattern recognition capabilities beyond your imagination. Now, my actual points. This method is not 100% bullet proof, you'd still get weird code, like an indexing variable held in a register for couple of tens of instructions, with disassembly you can do something about that, with binary code matching not so much, unless you in fact turn it into a disassembler. The second point is this: so what? (Don't read this as ball-breaking, but encouragement  ) I mean finding map boundaries is one problem and not a difficult one, finding what these maps actually are and make ME version independent is another thing. For my ECU I am practically interested in knowing around 400 maps (or single parameters / bits!) and sorting which one is which (eg. which KFPED is which, sport, reverse, automatic, manual gearbox, etc.). This I have not found a solution to yet, I mean one that would have considerable automation. This is what PRJ did in his map locator, but it requires manually crafted patterns and is not automatic in this respect (correct me if I am wrong). If I understand it correctly, it works well if you have two ECUs of the same family, you identify maps in one and created a pattern, then it will get you quickly to these maps in another ECU based on the same ME version. (No PRJ, I have not used your program, only looked through it once, so excuse me if I got it all wrong). |
|
|
|
|
Logged
|
|
|
|
360trev
Full Member
Karma: +68/-2
Offline
Posts: 235
|
|
« Reply #138 on: September 09, 2018, 08:02:47 AM »
|
|
|
I have some thoughts, because I actually more or less went all the way through this, using the same principle idea, but in a slightly different way. My very own private ST10 disassembler does that. Goes through the code once, finds all calls to scale reading routines, identifies scale index variables and the corresponding scale sizes, then goes through the code again and finds all maps. For scale integrated maps does the same thing, only in one pass. And yes, it does have an almost 100% hit rate. It even identifies double indirect referenced maps (very common in some of my bins, when m/a gearbox map is selected through an additional pointer reference). All is needed is to identify the address to the IROM block where the OS routines for map reading are. So far this part is manual, but can be easily automated too (I work with only one family of ECUs, so I don't really know how different it is on the other MEs). You can easily as well do it as an IDA script, I have chosen a different route because of he language I use for this, that has pattern recognition capabilities beyond your imagination. Now, my actual points. This method is not 100% bullet proof, you'd still get weird code, like an indexing variable held in a register for couple of tens of instructions, with disassembly you can do something about that, with binary code matching not so much, unless you in fact turn it into a disassembler. The second point is this: so what? (Don't read this as ball-breaking, but encouragement ) I mean finding map boundaries is one problem and not a difficult one, finding what these maps actually are and make ME version independent is another thing. For my ECU I am practically interested in knowing around 400 maps (or single parameters / bits!) and sorting which one is which (eg. which KFPED is which, sport, reverse, automatic, manual gearbox, etc.). This I have not found a solution to yet, I mean one that would have considerable automation. This is what PRJ did in his map locator, but it requires manually crafted patterns and is not automatic in this respect (correct me if I am wrong). If I understand it correctly, it works well if you have two ECUs of the same family, you identify maps in one and created a pattern, then it will get you quickly to these maps in another ECU based on the same ME version. (No PRJ, I have not used your program, only looked through it once, so excuse me if I got it all wrong). Thank you woj for an intelligent well thought out reply. I have actually just been writing a C167 dissassembler (as part of this just to become more familiar with the machine code, syntax and operations you can do with the various instructions) which I will also host into my github account shortly. I needed this to be able to 'walk' through the machine code sequences and find if they are 2 or 4 byte lengths as part of my scanning routines. IDA has a loverly abstracted interface to accomplish this via a plugin or IDA scripts (which I used initially to prove some concepts) but I want my work to be non incumbered by any commerical licensing requirements at all hence doing my own thing. As such I'm not interested in the slightest in using IDA scripting since in my (humble) opinion its also very useful to be able to do all of this work without utilizing ANY 3rd party tools which are commerical, no IDA, no WinOLS, etc. It should ideally be fully self contained with all the source-code available for inspection by the reader. Also makes it less likely to be broken by any updates Its all well and good keeping all this stuff private (and I absolutely respect your rights to do this) but in my situation where I am not making a living from any of this, its absolutely just a hobby I will open source all of my work as a help to others and so it can be improved on in the future or ported to other platforms. I'm not a huge fan of Java (hate it to be honest despite having had to use it before in some larger scale industry projects in a former life!) I just didn't want to use prj's map finder as I felt I wanted to take it in a different direction anyway. In terms of the so what points I have a clever way (which for now I will keep under wrap [need to have some surprises!) which I automatically identify the map names from. I've done extensive work on my own rom now and as such I've identified over 3,000 variables and functions (!) as well as documented the all the xref's back from variables to defining functions. This is a very powerful database which I hope to deploy, I've already done some preliminary work on this and optimistically I think I can get the majority of functions and variables in any ME7.x rom defined automatically. I'll keep you all posted as I progress the concept... |
|
|
|
|
Logged
|
|
|
|
360trev
Full Member
Karma: +68/-2
Offline
Posts: 235
|
|
« Reply #139 on: September 09, 2018, 08:06:49 AM »
|
|
|
I posted this before here... atic void PlugIn_process(int iArg)
{
char funcName[MAXSTR];
char filename[MAXSTR];
char mnem[MAXSTR];
char tmp[1024*512];
int offset=0;
int inst_count,func_len;
unsigned int x, num_funcs, num_segments;
ea_t addr, start_addr;
asize_t our_size;
flags_t flags;
func_t *f=0;
segment_t *seg=0;
// We are only interested in segments containing code.
seg = getnseg(0);
num_funcs = get_func_qty();
// Loop through each function
msg("Found <%d> functions\n",num_funcs);
for (x = 0; x < num_funcs; x++)
{
f = getn_func(x); // get ptr to the function itself
if(f == 0) {
msg("func not found at address %p\n");
}
else
{
// start address of this function..
start_addr = f->startEA;
addr = start_addr;
get_func_name(addr, funcName, sizeof(funcName)-1); // find the function name
// set_name(addr, new_funcName, SN_NOWARN);
flags = get_flags_novalue(addr);
msg("Found %-32.32s: at %p, func_flags(%lx)",funcName,addr,flags);
// Add name if there's no meaningful name assigned.
if(has_name(flags) != 0) { msg("(customized name ) "); }
if(has_dummy_name(flags) != 0) { msg("(autogenerated name) "); }
if(has_auto_name(flags) != 0) { msg("(has_auto_name ) "); }
//
// calc number of instructions
//
inst_count = 0;
addr = start_addr; // start of function
func_len = ((f->endEA)-(start_addr));
// count the number of instructions within the function
for (; addr < f->endEA;) {
// Get the flags for this address
flags = get_flags_novalue(addr);
// Only look at the address if it's a head byte, i.e. the start of an instruction and is code.
if (isHead(flags) && isCode(flags)) {
inst_count++; // increase number of instructions counter we have seenn within this function
}
our_size = get_item_size(addr); // get number of bytes within this function
addr += our_size; // move addres pc to next instruction...
}
//
// at this point 'inst_count' is total number of asm instructions within the buffer
//
msg(" func len = %-8d bytes (%-8d instructions)\n",func_len, inst_count );
sprintf(filename,"c:\\bin\\%p_%s_%d.bin",start_addr,funcName,inst_count);
#if 1
addr = start_addr; // start of function
offset=0;
// lets now loop through the instructions in each function
for (; addr < f->endEA;)
{
// Get the flags for this address
flags = get_flags_novalue(addr);
// get the size of this item (e.g. instruction length)
our_size = get_item_size(addr);
// Only look at the address if it's a head byte, i.e.
// the start of an instruction and is code.
if (isHead(flags) && isCode(flags))
{
char instruction[16];
// Fill the cmd structure with the disassembly of the current address and get the mnemonic text.
ua_mnem(addr, mnem, sizeof(mnem)-1);
ua_ana0(addr);
ua_mnem(addr, instruction, sizeof(instruction));
tag_remove(instruction, instruction, sizeof(instruction));
// lets view the mnemonic of *this* address
// msg("%p:[%-2.2d] ", addr, our_size);
#if 1
msg(".");
get_many_bytes(addr,&tmp[offset],our_size);
offset += our_size;
#else
for(int j=0;j < our_size; j++)
{
tmp[j] = get_byte(addr+j);
}
#endif
// show the hex dump of the instruction
// hex_dump((unsigned char *)&tmp[offset], our_size);
// show the mnemonic of the instruction
// msg(" %-8.8s\n",instruction);
}
// move addres pc to next instruction...
addr += our_size;
}
// dump it..
msg("dumping %s\n",filename);
save_file(filename, (unsigned char *)tmp, (size_t)offset);
#endif
}//end if
}//end for
} Using IDA's instruction walking interface this alone dumps every function in a ME7... I'm sure its easy to do with IDA scripting. If you just add the masking out the reloc's you have a very powerful way to cross reference functions across different roms with very little work.. |
|
|
|
|
Logged
|
|
|
|
|
woj
|
|
« Reply #140 on: September 09, 2018, 09:07:57 AM »
|
|
|
(PRJ is going to be thrilled to see all this ranting in his thread  ) Your motivation for not using IDA and such is more or less exactly as mine. In fact, the only commercial tool I use (and that is not even really commercial, just semi-closed source) is TunerPro, all of the other software I use is mine, including a flasher and a macro assembler to patch code. These are poor excuses to keep my software private, but there are some: 1. it is very targeted in its current form, so using it in a more general context than my ECU won't work. 2. Releasing things means it needs to have decent SE quality, my stuff does not and I don't have time to fix it. (It's so bad, that I cannot even trace my own way of thinking in software I have written half a year before). 3. Releasing things usually causes a shit-storm of stupid questions and I simply cannot deal with it. 4. Some software requires custom made hardware, and without it it's useless. So, what I do instead is I release bit and independent pieces of information here and there so that the inclined ones can make some use of it. I am curious about your approach, but reading "database" scares me a bit. A possible future direction to consider is to go beyond ST10 / ME, and develop something for TriCore for example |
|
|
|
|
Logged
|
|
|
|
|
prj
|
|
« Reply #141 on: September 09, 2018, 01:51:32 PM »
|
|
|
I mean rant on, why not I think ME7 is ... well sorta dead for this stuff. As in most stuff that exists is already decently defined or there is a hex/a2l for a similar firmware that can be found... |
|
|
|
|
Logged
|
|
|
|
|
woj
|
|
« Reply #142 on: September 09, 2018, 02:07:14 PM »
|
|
|
Well, that's one opinion and I partly agree with it. If one's a pro then defo the new stuff counts much more. But then, you get tons of new people getting on here every day crying for help with maps, LC/NLS patches, gargles/pops, and what not. As long as there will be cars with ME7 there will be non zero interest. From amateurs. (Myself, I seem to be always two generations of ECUs behind the mainstream).
Having said that, I'd really like to see cutting edge tools for newer architectures. I recently realised Fiat has swapped my beloved ME7.9.10 technology for my beloved Fiat t-jet engine with ME17.3.0 which is a whole new ball game (for me at least). One that I would eventually want to get into, just some fucker some time ago decided a day should have only 24 hours :/
|
|
|
|
|
Logged
|
|
|
|
|
eliotroyano
|
|
« Reply #143 on: September 10, 2018, 06:36:00 PM »
|
|
|
Maybe this kind of MAP locator tool can be tweaked for newer software and platforms. Of course if are not totaly encrypted or masked.
|
|
|
|
|
Logged
|
|
|
|
|
nyet
|
|
« Reply #144 on: September 12, 2018, 10:05:11 PM »
|
|
|
Also: detect ram locations to extend/replace ME7Info
|
|
|
|
|
Logged
|
ME7.1 tuning guideECUx PlotME7Sum checksumTrim heatmap toolPlease do not ask me for tunes. I'm here to help people make their own. Do not PM me technical questions! Please, ask all questions on the forums! Doing so will ensure the next person with the same issue gets the opportunity to learn from your ex
|
|
|
360trev
Full Member
Karma: +68/-2
Offline
Posts: 235
|
|
« Reply #145 on: September 21, 2018, 03:48:44 AM »
|
|
|
Also: detect ram locations to extend/replace ME7Info
Yes, absolutely agree on this one as it finds virtually nothing for a Ferrari rom and I want to do realtime telemetry for my track days so this is a must.. I've just been finishing my C16x/ST10 dissassembler which I will spin off into its own seperate project on github as a shared library and simple frontend for other tools to use within them. This is already working in basic format... -[ Basic Firmware information ]-----------------------------------------------------------------
>>> Scanning for ROM String Table Byte Sequence #1 [info]
found needle at offset=0x2cc04
found table at offset=00019A5C.
0X2CC04 [+0 ]: mov var_b, Rw_a [4] F6 F4 42 E2
0X2CC08 [+4 ]: mov var_b, Rw_a [4] F6 F5 44 E2
0X2CC0C [+8 ]: jnb cbitaddr_a, cbitaddr_c, Rel_b [4] 9A 22 05 10
0X2CC10 [+12 ]: movb Rb_a, #cdata16_b [4] E7 F8 11 00
0X2CC14 [+16 ]: movb var_b, Rb_a [4] F7 F8 0A E2
0X2CC18 [+20 ]: jmpr ccc_0000,Rel_l_a [2] 0D 04
0X2CC1A [+22 ]: movb Rb_a, #cdata16_b [4] E7 F8 14 00
0X2CC1E [+26 ]: movb var_b, Rb_a [4] F7 F8 0A E2
0X2CC22 [+30 ]: mov Rw_a, #cdata16_b [4] E6 F4 C8 CD
0X2CC26 [+34 ]: mov Rw_a, #cdata16_b [4] E6 F5 82 00
0X2CC2A [+38 ]: mov var_b, Rw_a [4] F6 F4 32 E2
0X2CC2E [+42 ]: mov var_b, Rw_a [4] F6 F5 34 E2
0X2CC32 [+46 ]: rets [2] DB 00
0X2CC34 [+48 ]: mov Rw_b, Rw_a [2] 88 60
0X2CC36 [+50 ]: mov Rw_a, #cdata16_b [4] E6 F4 86 2B
0X2CC3A [+54 ]: mov Rw_a, #cdata16_b [4] E6 F5 00 00
0X2CC3E [+58 ]: mov var_b, Rw_a [4] F6 F4 B2 E1
***
Idx=1 { 182542.000 } 0x101a1 : VMECUHN [Vehicle Manufacturer ECU Hardware Number SKU]
Idx=2 { 0261204841 } 0x1018b : SSECUHN [Bosch Hardware Number]
Idx=4 { 0000000000 } 0x10196 : SSECUSN [Bosch Serial Number]
Idx=6 { F131 CHALLENGE } 0x10177 : EROTAN [Model Description]
Idx=8 { R.BOSCH001 } 0x19a50 : TESTID
Idx=10 { 069117/15H52CE2 } 0x10167 : DIF
Idx=11 { 0691175H } 0x1015e : BRIF
>>> Scanning for EPK information [info]
found needle at offset=0x27902.
EPK: @ 0x10029 { /1/ME7.3/69/117/F131_US//15h52ce2/151299/ }
-[ PUKANS Air Pulsation correction Table ]-----------------------------------------------
Function byte-sequence found @ 0x4b8c2, needle offset is +44 bytes, seg offset is +48 bytes
0X4B8C2 [+0 ]: mov Rw_a, #cdata16_b [4] E6 FC E4 0A
0X4B8C6 [+4 ]: movbz Rb_a, cmm_b [4] C2 FD 6C F8
0X4B8CA [+8 ]: movbz Rb_a, cmm_b [4] C2 FE 7B F9
0X4B8CE [+12 ]: calls cseg_a, ccadr_b [4] DA 00 12 73
0X4B8D2 [+16 ]: movb var_b, Rb_a [4] F7 F8 51 8B
0X4B8D6 [+20 ]: movbz Rb_a, cmm_b [4] C2 F4 52 8B
0X4B8DA [+24 ]: movbz Rb_a, cmm_b [4] C2 F5 51 8B
0X4B8DE [+28 ]: mul Rw_a, Rw_b [2] 0B 45
0X4B8E0 [+30 ]: mov var_b, Rw_a [4] F6 07 CE 9D
0X4B8E4 [+34 ]: mov Rw_a, cmm_b [4] F2 F4 0E FE
0X4B8E8 [+38 ]: mov Rw_a, [Rw+] [2] 98 90
0X4B8EA [+40 ]: rets [2] DB 00
0X4B8EC [+42 ]: mov Rw_a, #cdata16_b [4] E6 FC AA 05
0X4B8F0 [+46 ]: mov Rw_a, #cdata16_b [4] E6 FD 06 02
0X4B8F4 [+50 ]: movbz Rb_a, cmm_b [4] C2 FE B8 8B
0X4B8F8 [+54 ]: calls cseg_a, ccadr_b [4] DA 82 78 97
0X4B8FC [+58 ]: movb var_b, Rb_a [4] F7 F8 D0 9D
0X4B900 [+62 ]: rets [2] DB 00
0X4B902 [+64 ]: mov Rw_a, cmm_b [4] F2 F4 2C 8F
0X4B906 [+68 ]: cmp ... [2] 48 40
***
PUKANS
Long identifier: Pulsation correction dependent on intake air temperature.
Display identifier:
Address: 0x8185aa
Value:
No. | 0 1 2 3 4 5 6 7
PHY| 11.00 37.00 64.00 91.00 104.00 117.00 131.00 171.00
--------------+------------------------------------------------------------------------
PHY| 1.0859 1.0391 1.0000 0.9688 0.9453 0.9375 0.9219 0.8828
Cells:
Unit:
Conversion name:
Conversion formula: f(phys) = 0.0 + 0.007812 * phys
Data type: UBYTE
X-axis:
Unit: Grad C
Conversion name:
Conversion formula: f(phys) = 0.0 + 1.000000 * phys
Data type: UBYTE
And I'm just finishing the bits required to make it into a proper disassembler and single line assembler. The idea is you can then upload patched code at runtime (during telem sessions) as well as realtime map editing. If you have a function uplad feature you can add features on the fly without having to reflash which for my usage is a must as I want to be able to rapidly develop and test without the whole reflash cycle which is just too time consuming... I needed the dissassembler so I could easily work on runtime/offline patching... Watch this space a big update is coming soon on my ME7 Swiss Army Knife tool |
|
|
|
|
Logged
|
|
|
|
360trev
Full Member
Karma: +68/-2
Offline
Posts: 235
|
|
« Reply #146 on: September 21, 2018, 03:52:46 AM »
|
|
|
Maybe this kind of MAP locator tool can be tweaked for newer software and platforms. Of course if are not totaly encrypted or masked.
Yes, I've already acquired a Hitex Shield Buddy which boasts a fully programmable Infineon TriCore TC275 with full SDK and compilers, etc. I will get to this in good time.. |
|
|
|
|
Logged
|
|
|
|
taco
Newbie
Karma: +0/-0
Offline
Posts: 4
|
|
« Reply #147 on: December 27, 2018, 07:35:37 AM »
|
|
|
The tool is amazing. I used nyet's fork and it found a nice shortlist to add to my "in progress" def.
ME7Logger can monitor many different items....
If the source is available maybe something from it can be adapted to help the map finder?
|
|
|
|
|
Logged
|
|
|
|
zarboz
Newbie
Karma: +2/-0
Offline
Posts: 12
|
|
« Reply #148 on: March 11, 2019, 09:59:10 AM »
|
|
|
I am having a slight issue with the software where the [Me7XmlPlugin] will print the proper hex offset but the xdf / UI does not display the proper hex address It is either how the app is temporarily storing the offset when identifying the addr or its the string.format formatting an existing real hex offset into a null hex offset Terminal shows: [Me7XmlPlugin] Pattern for CLRHK found at: 0x782E8 XDF export shows <title>CLRHK</title> <XDFAXIS id="z"> <decimalpl>2</decimalpl> <EMBEDDEDDATA mmedtypeflags="0x0" mmedaddress="0x27F2" mmedelementsizebits="8"/> <MATH equation="0.000000+X*1.000000"> <VAR> <id>X</id> </VAR> </MATH> </XDFAXIS> and UI also shows 27F2 but real offset is : 0x782E8 as reported by the initial Me7XmlPlugin which oddly enough uses the same string.format command if anyone can help me out it would be awesome TLDR: Me7XmlPlugin reports proper offset in terminal but UI / XDF show completely different offset my branch is here: https://github.com/zarboz/me7-tools |
|
|
|
« Last Edit: April 02, 2019, 07:13:59 AM by zarboz »
|
Logged
|
|
|
|
marcjero
Full Member
Karma: +4/-1
Offline
Posts: 60
|
|
« Reply #149 on: July 19, 2022, 05:59:13 PM »
|
|
|
Hi, I have some trouble implementing static axes with the tool. My TVUB axis is calculated so it's not possible to manage it as a table. On top of that the axis definition is located just before the TVUB table and code is referencing the axis definition. The axis defintion is 6 bytes long, so basically I have to add a 6 bytes offset to the final address in order to locate the TVUB. That's an idea I got to solve that: 1/ allow to define static content for maps, so we could define static axes this way 2/ Define the TVUB_AXIS with static content and then declare TVUB as following the TVUB axis I don't know if Tunerpro allows to define tables with data stored in the XDF file (why would it do that really?) But one can define axis values in the XDF file for sure. Therefore the XDF export could filter out table defintions that contains data and populate axis values instead of using axis references. What do you think about this proposal ? here is the A2L defintion of the TVUB axis: /begin CHARACTERISTIC
TVUB "EV-Spannungskorrektur"
CURVE 0x1466C DAMOS_FKL1 0 UTS 0.000000 10.240000
/begin AXIS_DESCR
FIX_AXIS UBATT UBQ 17 0.000000 26.037190
FIX_AXIS_PAR 0 4 17
EXTENDED_LIMITS 0.000000 26.037200
/end AXIS_DESCR |
|
|
|
|
Logged
|
|
|
|
|
