NefMoto

Technical => Flashing and Chipping => Topic started by: n0ble on August 10, 2014, 03:10:22 AM



Title: FRF and SGO - Differences?
Post by: n0ble on August 10, 2014, 03:10:22 AM
For Vas-PC - Why are their 2 types of file?

As a guess, SGO is full flash and FRF is partial?

I have the stock files:
MED17.5.5 - MED1755_03C906027AD_1037518220
DQ200 - HW: 0AM927769D - v069E5110AM___getriebe_DSG_tbE5

Would i be right in thinking that the attached is the relevant update files for stock?
MED17.5.5 - FL_03C906027AD_9971.frf
DQ200 - v069E5610AM___getriebe_DSG_tbE5_sw.sgo

Also, can someone point me in the right direction of converting the DSG .SGO to a .BIN
AND
The MED17 .FRF to a .ODX?

I understand that i will not be able to update with the .FRF file using VAS-PC until it has been converted to .ODX?

Thanks again, and sorry for not be very clear i'm just a little lost with this.

Thanks



Title: Re: FRF and SGO - Differences?
Post by: TCSTigersClaw on August 18, 2014, 08:10:24 AM
this is an Ibiza FR file right ? the 9971 is almost the latest update (now it is 9972) , it is update only and VAS can write in OTP areas also.


Title: Re: FRF and SGO - Differences?
Post by: TCSTigersClaw on August 18, 2014, 08:14:27 AM
sorry forgot about the question.

frf and sgo are encrypted for VAS , I dont think they can be converted to bin. The best chance is to flash them via VAS and then read them via obd or tricore boot to get the BIN


Title: Re: FRF and SGO - Differences?
Post by: IamwhoIam on August 19, 2014, 07:42:33 AM
this is an Ibiza FR file right ? the 9971 is almost the latest update (now it is 9972) , it is update only and VAS can write in OTP areas also.

NO TOOL can write to OTP areas!!!! EVER!


Title: Re: FRF and SGO - Differences?
Post by: k0mpresd on August 19, 2014, 08:31:46 AM
sgo can be converted to bin.


Title: Re: FRF and SGO - Differences?
Post by: coreyj03 on August 19, 2014, 09:06:52 PM
i posted up the factory mk5 DSG launch control .sgo a couple years if u need it.  search its somewhere on here


Title: Re: FRF and SGO - Differences?
Post by: ddillenger on August 19, 2014, 09:29:21 PM
NO TOOL can write to OTP areas!!!! EVER!

Bullshit.

The first time they're easy to write.

It's the subsequent writes that aren't so easy.

:P



Title: Re:
Post by: n0ble on August 20, 2014, 08:26:00 PM
TCStigersclaw yes it is the Ibiza FR :-) where can I get the 9972 file? Latest I could find was 9971. I successfully flashed the 9971 file with ODIS engineering as vas-pc does not support the protocol the ECU is running on.

Vas-pc did work ok for flashing the DSG though.

ddillenger that is very true :-) although I'm lead to believe that the cks are in the OTP areas? So if the map data changes in a update file how do they deal with the CKS, as the previous CKS in the OTP will now be incorrect?


Title: Re: FRF and SGO - Differences?
Post by: k0mpresd on August 21, 2014, 11:01:19 AM
here is sgo file description, along with otp area for same ecu.

Quote
Reading ECU data, please wait...
PROCESSOR TYPE: TC1796
IROM FLASH sector configuration:
Address   Size    Access
A0000000h   4000h   read and write
A0004000h   4000h   read and write
A0008000h   4000h   read and write
A000C000h   4000h   read and write
A0010000h   4000h   read only (OTP)
A0014000h   4000h   read only (OTP)
A0018000h   4000h   read and write
A001C000h   4000h   read and write
A0020000h   20000h   read and write
A0040000h   40000h   read and write
A0080000h   80000h   read and write
A0100000h   80000h   read and write
A0180000h   80000h   read and write


Title: Re:
Post by: n0ble on August 22, 2014, 01:28:20 AM
Thanks k0mpresd.


Title: Re: FRF and SGO - Differences?
Post by: gremlin on August 24, 2014, 12:19:02 PM
sgo can be converted to bin.

No problem convert .FRF to bin also.
Just convert FRF to ODX (the same as XML-format file) and than extract BIN content (coded as BCB) from ODX.
As example below is ODX converted from frf-file given in topic start message.


Title: Re: FRF and SGO - Differences?
Post by: chli1976 on August 24, 2014, 09:39:23 PM
Just convert FRF to ODX (the same as XML-format file) and than extract BIN content (coded as BCB) from ODX.

Can you give more infos
thanks


Title: Re: FRF and SGO - Differences?
Post by: tabster on December 26, 2014, 05:51:24 PM
For Vas-PC - Why are their 2 types of file?

As a guess, SGO is full flash and FRF is partial?


SGO files are for ECUs using KWP2000 protocol.
FRF, ODX and SOX files are for ECUs using UDS protocol.

In theory all types can be converted to bin, however many different methods of compression and encryption exist, so it can take some time and effort to do it.



Title: Re: FRF and SGO - Differences?
Post by: chli1976 on December 27, 2014, 03:14:17 AM
coded as BCB

Is this the right method

http://timogruss.de/2014/01/bcd-in-dezimalwerte-umrechnen/#BCD_zu_Dezimal_umrechnen


Title: Re: FRF and SGO - Differences?
Post by: technic on December 27, 2014, 06:29:53 AM
BCB, not BCD :) BCB is a compressed format delevoped by Bosch


Title: Re: FRF and SGO - Differences?
Post by: Aurélien on December 27, 2014, 03:29:22 PM
Bosch ODX content is COMPRESSED and ENCRYPTED ( 11 )

Encryption is very easy.
Decompression is easy also... :)
Compression, the proper way ( not just tellng " following block is uncompressed " ) is a lot more work though.


Title: Re: FRF and SGO - Differences?
Post by: KmosK04 on January 16, 2015, 07:48:54 AM
Can somebody know how to convert .frf files to .bin? I have an app that converts them to .odx. Now I have to convert that to .bin? If yes how?? Thanks


Title: Re: FRF and SGO - Differences?
Post by: KmosK04 on January 19, 2015, 02:07:04 PM
Anyone please?


Title: Re: FRF and SGO - Differences?
Post by: dream3R on October 09, 2015, 03:29:27 PM
So the resulting bin from FRF hat does it all contain?


Title: Re: FRF and SGO - Differences?
Post by: nyet on October 09, 2015, 07:02:47 PM
Bosch ODX content is COMPRESSED and ENCRYPTED ( 11 )

Encryption is very easy.
Decompression is easy also... :)
Compression, the proper way ( not just tellng " following block is uncompressed " ) is a lot more work though.


Too bad nobody has the balls to release source code.

It is easy to say something is "easy".

It isn't easy to document and publish.

All balless wonders who talk a lot but not much else.


Title: Re: FRF and SGO - Differences?
Post by: n0ble on October 10, 2015, 01:17:41 PM
I'm almost there with it....

Now at the final step of trying to work out the compression, I have half worked out the compression but unfortunately my knowledge lacks here.

However i'll keep at it, I'll get there in the end.


Title: Re: FRF and SGO - Differences?
Post by: dream3R on October 11, 2015, 10:56:45 AM
Anyone know if VAG use FRF for UDS definitions i.e.  ReadDataByIdentifier?



Title: Re: FRF and SGO - Differences?
Post by: dream3R on October 11, 2015, 12:10:41 PM
I'm almost there with it....

Now at the final step of trying to work out the compression, I have half worked out the compression but unfortunately my knowledge lacks here.

However i'll keep at it, I'll get there in the end.

Care to post your progress?   I was looking before but didn't look like something i've seen before.

edit:  keeping wih it, assuiming it's within you, it's the best thing.  I nearly wen't mad doing 5 bar on my Volvo but got there.  BTW the one on here is incomplete...mods know this etc, prj knows it, pisses me off!



Title: Re: FRF and SGO - Differences?
Post by: dream3R on October 11, 2015, 12:12:50 PM
Bosch ODX content is COMPRESSED and ENCRYPTED ( 11 )

Encryption is very easy.
Decompression is easy also... :)
Compression, the proper way ( not just tellng " following block is uncompressed " ) is a lot more work though.


Come-on then friend, give me some clues, you were happy enough for my FREE MED9 help!


Title: Re: FRF and SGO - Differences?
Post by: Geremia on October 17, 2015, 03:59:52 AM
frf-to-odx is done inside SoxUtil.dll (odis or DTS7)
Code:
text:10001CC5                 call    edi ; MString::operator char const *(void) ; MString::operator char const *(void)
.text:10001CC7                 push    ebx             ; dest_zip_filename
.text:10001CC8                 push    eax             ; frf_filename
.text:10001CC9                 lea     ecx, [ebp+var_170]
.text:10001CCF                 call    MY_getKey_and_goto_descramble
.text:10001CD4                 lea     ecx, [ebp+var_170]
.text:10001CDA                 mov     byte ptr [ebp+var_4], 2
.text:10001CDE                 call    MY_unzipper_stuff
.text:10001CE3                 test    al, al
.text:10001CE5                 jnz     short loc_10001CFC
.text:10001CE7                 lea     ecx, [ebp+var_28]
.text:10001CEA                 push    offset unk_100046E0
.text:10001CEF                 push    ecx
.text:10001CF0                 mov     [ebp+var_28], offset aNotOne_odxInAr ; "Not one .odx in archive"
.text:10001CF7                 call    _CxxThrowException

key.bin is inside the resource area

Then, as told, the odx contains flash data in encrypted/compressed form.
I dont' know for ECUs, you need to RE bootarea to know the decryption/decompression algo and i did only for dq200 0CW, and yes they are simple once ported to C code, but takes some days to RE them, so i'm not surprised if they don't go opensource quickly.
Flashdata first need to be descrambled and, at least for dsg, it's the same scrambling algo found in previous sgo files, just the byte subst table is per ecu type.
About compression algo, don't know, probably dsg uses diff algo than bosch, anyway comparing compressed and uncompressed data makes the task very easy.


Title: Re: FRF and SGO - Differences?
Post by: tmbinc on November 09, 2015, 04:00:22 PM
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

const unsigned char key[4095] = {
#include "key.h"
};

int main(void)
{
   int kidx = 0;
   int seed0 = 0;
   int seed1 = 1;
   
   while (1)
   {
      unsigned char buf[1024];
      int i;
      int n = read(0, buf, sizeof(buf));
      if (!n) {
         break;
      }
      
      for (i = 0; i < n; ++i) {
         unsigned char kb = key[kidx];
         kidx += 1;
         kidx %= sizeof(key);

         seed0 = ((seed0 + kb) * 3) & 0xFF;

         buf ^= seed0 ^ 0xFF ^ seed1 ^ kb;

         seed1 = ((seed1 + 1) * seed0) & 0xFF;
      }
      
      write(1, buf, n);
   }
   return 0;
}


Title: Re: FRF and SGO - Differences?
Post by: Geremia on November 10, 2015, 03:45:36 PM
welcome! ;)
I do like this, but it's ok anyway, since keysize is 0xFFF
buf ^= seed0 ^ sizeof(key) ^ seed1 ^ kb;


Title: Re: FRF and SGO - Differences?
Post by: tmbinc on November 13, 2015, 05:16:18 PM
Also, to unpack "BCB Type1" compressed data:

import sys, struct

key = "BiWbBuD101"

img = sys.stdin.read()
img = img[img.index("\x1A\x01") + 2:]
img = ''.join(chr(ord(j)^ord(key[i%len(key)])) for i, j in enumerate(img))

p = 0
res = ""

while p < len(img):
  l = struct.unpack(">H", img[p:p+2])[0]
  p += 2

  fl = l >> 14
  l &= 0x3FFF

  if fl == 0: # literal
    res += img[p:p+l]
    p += l
  elif fl == 1: # RLE
    res += img[p] * l
    p += 1
  else:
    sys.stderr.write("remaining bytes: " + img[p:].encode('hex') + "\n")
    break

sys.stdout.write(res)



Title: Re: FRF and SGO - Differences?
Post by: Geremia on November 15, 2015, 05:11:44 PM
else //fl==3 checksum
      {
         p++;
         unsigned int chk=(inbuf[p]<<24)|(inbuf[p+1]<<16)|(inbuf[p+2]<<8)|inbuf[p+3];
         unsigned int chk2=0;
         for(unsigned int i=0;i<outsize;i++) chk2+=outbuf;
         if(chk!=chk2)
         {
            printf("Checksum mismatch, file=0x%X calc=0x%X at inbuf offset 0x%X\n", chk, chk2, p);
            error=true;
         }
         if((p+4)!=size)
         {
            printf("Checksum at offset 0x%X not EOF\n",p);
            error=true;
         }
         break;
      }


Title: Re: FRF and SGO - Differences?
Post by: H2Deetoo on November 16, 2015, 12:35:16 AM
This looks very interesting guys!

Is somebody able to post a complete example of input/output data to verify the posted routines?
I am interested in writing a (Delphi) application for this...


Thanks,
H2Deetoo


Title: Re: FRF and SGO - Differences?
Post by: DrDelphi on September 27, 2016, 11:42:38 PM
Hello guys

Too bad this thread has died.
If the members that posted the above code pieces are still around, I would have some questions if they are kind enough to clear them with me.
So, first of all, where do I get the "key" from ?
Normally, having a SGO and the BIN file, should be enough to find the key, but the seed algo doesn't check. Are you sure it is seed0 = (seed0 + key[kIdx]) * 3 ? Isn't it *2 or *4 or anything else?
The password encrypted SGOs are easier to decrypt. Some passwords I found are BiWbBuD101, GEHEIM, CodeRobert and MILKYWAY.

This should be all for now.
Looking forward for you replies.
Best regards.


Title: Re: FRF and SGO - Differences?
Post by: Geremia on September 30, 2016, 11:00:34 AM
The seed0 = (seed0 + key[kIdx]) * 3 you refer, is about decrypting frf to a zip file, inside zip there is odx file, inside odx you find DATA, which is (mont time) encrypted/compressed data you send (as is) to ecu when flashing.
In sgo, if i remember well, is a simply xor FF (or sort of) to get an sgm file, which is a container where you find DATA (again, most time compressed and/or encrypted) to send (as is) to ecu.

If you are referring to seed in the mean of seed/key auth against ecu, that's another story, SA2 data (you can find it inside sgm and odx) is what you need for passing seedkey in programming session (diag session is another story).

I came accross the milkyway too  ;)


Title: Re: FRF and SGO - Differences?
Post by: cherry on September 30, 2016, 12:58:11 PM
So far i know DATAblocks in sgm should only be "encrypted" as base64.


Title: Re: FRF and SGO - Differences?
Post by: cherry on September 30, 2016, 01:13:52 PM
Attached sgm example.


Title: Re: FRF and SGO - Differences?
Post by: DrDelphi on October 01, 2016, 01:17:50 AM
Thanks for the reply, Geremia.

I thought that the algo you posted was for decoding SGO blocks. Meanwhile I realized it was for FRF as I succeeded to turn them into ODX and then into BINs.

The problem I am dealing with now is with the SGOs.

The SGO files that are compressed, are also password encrypted and they are easy to decompress / decode. If you don't know the password, but you have the BIN from another source, I can find out the password by compressing the BIN and then XORing the result with the SGO.

The problem is with the uncompressed SGOs that don't use passwords and I couldn't yet figure out how they are encrypted.
If I take a SGO, extract its blocks, then take the corresponding BIN from another source and XOR them, I get some scrambled data. Then if I take the scrambled data and XOR it with another SGO's blocks, I get the correct data result, but it's not a solution I can rely on. It's definitely another XOR algo, but not the one used for FRFs and maybe another key too.

Any hint ?

Thanks in advance and best regards.


Title: Re: FRF and SGO - Differences?
Post by: Geremia on October 01, 2016, 03:11:15 AM
Well, sgo/odx are only containers, how DATA is compressed/encrypted is ecu and/or car brand specific, you have to reverse the service36 function of the bootloader of the specific ecu to know how it's done, or if you have some encrypted and decrypted pairs, maybe guess it.


Title: Re: FRF and SGO - Differences?
Post by: prj on October 05, 2016, 03:51:29 PM
The problem is with the uncompressed SGOs that don't use passwords and I couldn't yet figure out how they are encrypted.

If it's not encrypted then it's XOR FF and result is the file obviously, as has been posted here.
If it's BCB compressed XOR encrypted - then breaking the key takes <1 second with a single thread for any file, you don't need to know what's in there nor do you need the key, as it can be computed at runtime.
If it's some different type of encryption and/or compression, then you will have to obviously reverse the bootloader of the ECU to find the algorithm.

EVC has introduced a SGO/FRF import plugin. Upon asking them which ECU's it actually works with I did not receive a reply... and I am a customer.


Title: Re: FRF and SGO - Differences?
Post by: ladan on November 09, 2016, 11:27:11 AM
Hi everyone...  so where can I find "key" array?  Geremia ?

const unsigned char key[4095] = {
#include "key.h"
};


Title: Re: FRF and SGO - Differences?
Post by: nyet on November 09, 2016, 12:11:27 PM
If somebody wants to give me a set of sgo/bin pairs and decent documentation i'd be more than happy to write a decoder and provide source code.


Title: Re: FRF and SGO - Differences?
Post by: prj on November 10, 2016, 07:59:14 AM
If somebody wants to give me a set of sgo/bin pairs and decent documentation i'd be more than happy to write a decoder and provide source code.
Did you ignore what I wrote in the other thread?

I am bumping thic topic.
What's needed to write such converter? I would donate, others would do the same I guess.
1. Reverse SGO format enough to parse it.
2. Reverse the bootloader on every single ECU type you want to convert.

EDC15/ME7/ZF6HP/MED9/ME17 are BCB/XOR. Some have fixed key, others like EDC15 have a different key for every ECU file and version.
Other ECU's such as EDC16 and SIMOS have a completely different algorithm that can be only obtained by reversing the the bootloader.

I would happily write a converter if somebody can post the specs to the sgo file.
With all due respect, I don't think you are going to break the crypto on EDC16 for example. You need to pull the bootloader apart one by one, it is a very time consuming process. The SGO is easy to reverse even without any specs.

I don't think you understand how this works.
The code to decode ME7 is already in this forum thread, but it is different for every ecu type.


Title: Re: FRF and SGO - Differences?
Post by: ladan on November 11, 2016, 03:48:25 AM
hi Prj,
So what about my question? Don't you know? 
As far as I see - all "converters"(FRF 2 ODX/SOX) use original .dll (from DTS7) to convert FRF to ODX, but I searching algo(my solutions are running under Linux).... One posted above looks good but where to find a "key"?   


Title: Re: FRF and SGO - Differences?
Post by: prj on November 11, 2016, 05:06:57 AM
hi Prj,
So what about my question? Don't you know? 
As far as I see - all "converters"(FRF 2 ODX/SOX) use original .dll (from DTS7) to convert FRF to ODX, but I searching algo(my solutions are running under Linux).... One posted above looks good but where to find a "key"?   

Why do you feel entitled to an answer? It is not my job to answer you.
I have not even looked at the FRF format, I only wrote an SGO decoder and I had help with reversing the algorithms in some of the ECU's.


Title: Re: FRF and SGO - Differences?
Post by: ladan on November 11, 2016, 05:21:20 AM
Hm...  I just hoped you can help, since you are online....


Title: Re: FRF and SGO - Differences?
Post by: ladan on November 12, 2016, 06:57:35 AM
turned on my brain, read once again, found all I need, wrote unpacker script .... thanks Geremia, tmbinc


Title: Re: FRF and SGO - Differences?
Post by: learning1 on November 19, 2016, 11:00:44 AM
NO TOOL can write to OTP areas!!!! EVER!

That is incorrect.
OTP is write once area that starts with all binary bits unset value 00
When all binary bits are burned you have FF
You can only burn more bits so you can only move towards FF,

Tools tend to avoid OTP but the statement that no TOOL can write to this EVER is not TRUE


Title: Re: FRF and SGO - Differences?
Post by: IamwhoIam on November 22, 2016, 03:46:42 AM
That is incorrect.
OTP is write once area that starts with all binary bits unset value 00
When all binary bits are burned you have FF
You can only burn more bits so you can only move towards FF,

Tools tend to avoid OTP but the statement that no TOOL can write to this EVER is not TRUE

My bad, let me rephrase that: no OBD flash tool can EVER write OTP areas that have been programmed and set as OTP. EVAR.


Title: Re: FRF and SGO - Differences?
Post by: dera on March 16, 2017, 10:12:32 PM
My bad, let me rephrase that: no OBD flash tool can EVER write OTP areas that have been programmed and set as OTP. EVAR.

Yes they can, just that they can only go one way with it.
Let me rephrase your comment :) No tool can 0 OTP bits that have been set.


Title: Re: FRF and SGO - Differences?
Post by: eliotroyano on October 14, 2017, 07:56:44 PM
Friends what are main differences between SGM and SGO files?


Title: Re: FRF and SGO - Differences?
Post by: wangyanjun on April 06, 2018, 07:22:58 PM
https://avdi-forum.de/avdi_aktuell/index.php/Thread/581-VAG5054-SGO-file/

There is a tool used to extract SGO file. Who can get it and post it here?


Title: Re: FRF and SGO - Differences?
Post by: superglitch on April 06, 2018, 07:55:42 PM
It's only for old stuff.


Title: Re: FRF and SGO - Differences?
Post by: IamwhoIam on April 07, 2018, 06:02:27 AM
lol I don't think Revo internal tools were ever meant to become public domain.


Title: Re: FRF and SGO - Differences?
Post by: superglitch on April 07, 2018, 02:52:50 PM
lol I don't think Revo internal tools were ever meant to become public domain.

Same could be said about many pieces of software that we interact with.


Title: Re: FRF and SGO - Differences?
Post by: nyet on April 07, 2018, 11:01:38 PM
lol I don't think Revo internal tools were ever meant to become public domain.

I can count on one hand the number of people who publish their code in this so called "community".


Title: Re: FRF and SGO - Differences?
Post by: Jerin on November 22, 2018, 10:04:03 PM
frf-to-odx is done inside SoxUtil.dll (odis or DTS7)
...
key.bin is inside the resource area

I see that soxutil.dll calls a function with a "\\key.bin" entry but no file exists.
Is the resource inside the .dll?
.rsrc/1033/FILE/1002 holds 4k of binary info, is this it?

Thank you


Title: Re: FRF and SGO - Differences?
Post by: Jerin on December 22, 2018, 10:49:15 AM
Still looking for an answer for this.
I can't find any resource linked with key.bin.
How about a hint if you can.


Title: Re: FRF and SGO - Differences?
Post by: Jerin on January 02, 2019, 11:34:23 AM
Also, to unpack "BCB Type1" compressed data:

import sys, struct

key = "BiWbBuD101"

img = sys.stdin.read()
img = img[img.index("\x1A\x01") + 2:]
img = ''.join(chr(ord(j)^ord(key[i%len(key)])) for i, j in enumerate(img))

p = 0
res = ""

while p < len(img):
  l = struct.unpack(">H", img[p:p+2])[0]
  p += 2

  fl = l >> 14
  l &= 0x3FFF

  if fl == 0: # literal
    res += img[p:p+l]
    p += l
  elif fl == 1: # RLE
    res += img[p] * l
    p += 1
  else:
    sys.stderr.write("remaining bytes: " + img[p:].encode('hex') + "\n")
    break

sys.stdout.write(res)



I get:
    l = struct.unpack(">H", img[p:p+2])[0]
TypeError: a bytes-like object is required, not 'str'

I used an odx file created by VAS File Decoder.
This is in the source:
[FORMAT]
BCB Type1 (C) R. Bosch GmbH 2000 K3/EMW3-Ws
[START_ADDRESS]
80020000
[END_ADDRESS]
8027FFFF



Title: Re: FRF and SGO - Differences?
Post by: nyet on January 02, 2019, 11:41:14 AM
you might need an 'img=bytearray(img)' somewhere to convert the string to a byte array


Title: Re: FRF and SGO - Differences?
Post by: Jerin on January 02, 2019, 01:43:38 PM
Thanks for the idea.

I placed the line just before the 'while'.
I got a TypeError because didn't include the "optional" encoding. Hmm?
So I:
img = bytearray(img, 'utf_8')
also tried utf_16 and ascii

Each fail when it hits a char it can't encode.
Too bad I can't:
img = bytearray(img, 'hex')

I am not entirely sure that chr() here:
img = ''.join(chr(ord(j)^ord(key[i%len(key)])) for i, j in enumerate(img))
can handle the extended char set (>127)

But I don't know python.




Title: Re: FRF and SGO - Differences?
Post by: nyet on January 02, 2019, 02:29:13 PM
maybe something like
Code:
from array import array

img = array("B", img)

you might have to learn python for this.. if i have time I might try to fix it for you but i'll need the stuff you're working with


Title: Re: FRF and SGO - Differences?
Post by: Jerin on January 02, 2019, 09:21:22 PM
.. if i have time I might try to fix it for you ..

Very kind of you but that would use too much of your time.
I can't even be sure of the script's efficacy.
I will work on my understanding of Python.


Title: Re: FRF and SGO - Differences?
Post by: Jerin on January 03, 2019, 02:39:26 PM
So I tested a small script with a similar struct.unpack:
Code:
import struct
h = 'Hi'
s = struct.unpack('>H', h)[0]
print s
and I got the same error as above.  Online this script works fine.

Conclusion- the script here that I am having the issue with is programmed for Python version 2.  It will need to be re-worked for version 3.

My lazy (smart) fix was to uninstall Python3 and install Python2.


Title: Re: FRF and SGO - Differences?
Post by: Jerin on January 07, 2019, 10:52:55 PM
My odx has 5 <DATA> sections:
<DATA> section 3
 80000000-80003FFF
 3FFF bytes = 16383 dec

<DATA> section 1
  80004000-8000FEFF
  BEFF bytes = 48895 dec

<DATA> section 2
  80020000-8027FFFF
  25FFFF bytes = 2490367 dec

<DATA> section 4
  80283000-8037FFFF
  FCFFF bytes = 1036287 dec

<DATA> section 5
  80380000-803FFEFF
  7FEFF bytes = 524031 dec

I have labelled the sections from 1 to 5 but section 3 is defined in the header as the lowest hex location.
Running the "unpack BCB Type1 compressed data" script on each section leaves many remaining bytes and none produce the needed 4 MB binary.
Stacking the sections in 12345 order and running through the script produced a very small output but no remaining bytes.
The 31245 order was nearly the same.

Please, how is this supposed to work?

A positive note, section3 unpacked does show the actual SW number related to the frf.


Title: Re: FRF and SGO - Differences?
Post by: kuebk on January 08, 2019, 01:41:35 PM
FRF/SGO might not give you full binary, some sections might be missing.


Title: Re: FRF and SGO - Differences?
Post by: Jerin on January 19, 2019, 09:25:55 PM
My odx has 5 <DATA> sections:
<DATA> section 3
 80000000-80003FFF
 ..
<DATA> section 1
  80004000-8000FEFF
 ..etc


Ok, data sections show the data locations, just fill the empty sections with 00 00 00..
up to the final file size 3fffff (first byte is at 0).
No extra checksum to do.


Title: Re: FRF and SGO - Differences?
Post by: birchbark506 on January 20, 2019, 06:34:55 AM
can a sgo be modded and then reflash back using odis? i am running in to a problem with cmd on med17 with tp20 tuning protection i have odis to flash the modded sgo back with


Title: Re: FRF and SGO - Differences?
Post by: cherry on January 21, 2019, 10:52:22 AM
Most expensive tool and struggling with a TP20 ecu, buy PCMflash for these ecu...

To the question, it doesnt make sense to pack it into a sgo. The problem is not to write it but the RSA signature. So either you bypass it via OBD unlock or patch flash via bootmode, then you can write OBD. Only ecu lower TP07 can be written via OBD without "tprot"-bypass. This is very old stuff.


Title: Re: FRF and SGO - Differences?
Post by: Jerin on January 21, 2019, 11:43:01 AM
.. it doesnt make sense to pack it into a sgo. The problem is not to write it but the RSA signature.

Won't Odis know how to get past the RSA?  Probably why he wants to repack.

Quote
So either you bypass it via OBD unlock ..

So absolutely no need to open the ECU even once?

Quote
..via OBD without "tprot"-bypass.

I though that Tprot is different from RSA.  You need to descramble a random RSA passkey to read or write to the ECU but if the Tprot bytes are set, you can't dump the tuner's info.
Have I got this right?



Title: Re: FRF and SGO - Differences?
Post by: birchbark506 on January 21, 2019, 01:41:16 PM
i wish cmd would update it and give a VR read to be able to read and write ecu with RSA code, seems like all the med17 tp in canada are locked dose pcmflash give VR and checksum before flashing back to ECU


Title: Re: FRF and SGO - Differences?
Post by: cherry on January 21, 2019, 06:37:46 PM
Odis just write content what´s inside the sgo/sgm, there is no RSA during flash, it´s done from Bosch. Also there is RSA in older MEDC17 or even EDC16, but it was not checked hard enough, so a calculation was possible.


Title: Re: FRF and SGO - Differences?
Post by: d3irb on February 20, 2019, 02:26:12 PM
Hi - I posted a reply to the Simos18 thread earlier but I have reversed the Simos18 encryption and compression setup so I can convert Simos18 ODX to BIN.

Please find my Python3 script attached to http://nefariousmotorsports.com/forum/index.php?topic=10364.msg122889#msg122889 .

A few things relevant to this thread:

1) "BCB Type1" is basically "LZSS" compression. I adapted an LZSS library from elsewhere and the code is much cleaner than what has been posted in this thread. My script contains the LZSS decompression routine which should work for your other ECUs. This code was modified from https://github.com/magical/nlzss

Code:
def decompress_raw_lzss10(indata, decompressed_size):
    """Decompress LZSS-compressed bytes. Returns a bytearray."""
    data = bytearray()

    it = iter(indata)

    def writebyte(b):
        data.append(b)
    def readbyte():
        return next(it)
    def readshort():
        # big-endian
        a = next(it)
        b = next(it)
        return (a << 8) | b
    def copybyte():
        data.append(next(it))

    while len(data) < decompressed_size:
        b = readbyte()
        flags = bits(b)
        for flag in flags:
            if flag == 0:
                copybyte()
            elif flag == 1:
                sh = readshort()
                count = (sh >> 10)
                disp = (sh & 0x3ff)
                for _ in range(count):
                    writebyte(data[-disp])
            else:
                raise ValueError(flag)

            if decompressed_size <= len(data):
                break
    return data

2) I see a lot of confusion here. To be crystal clear: ODX is simply an XML file which contains the payload for ODIS to send to the ECU. Each ECU has its own system for decrypting, decompressing, and flashing, which differs from ECU to ECU. While some things are shared (like this compression type), other things (like the AES keys for Simos18, which you can find in my script) are different from ECU to ECU. Because ODIS does not need to flash modified maps, it is not as capable as third-party/aftermarket flashing tools - it never needs to do anything like "fix checksums" or "RSA" because the files it is flashing are already valid and come from VW.

For example, "RSA protection" is a check which exists only inside the ECU - ODIS knows nothing about it. When the ECU boots it attempts to verify the RSA signature of the various protected areas on its own. In the case of the ECUs which ME7Sum can fix, a foolish property of VW's protection is exploited: the flash routines in the ECU don't check the RSA signature against the currently running code, so both the signatures and the public key can be replaced so long as they match ("self-signing" the flash). The flash routine will happily write the file and the file itself will happily verify because it's been signed with its own key. For some newer ECUs, signatures are checked by the flash routine so this is no longer possible.


Title: Re: FRF and SGO - Differences?
Post by: pechspils on July 07, 2019, 05:59:17 AM
I've used the information posted in this thread to write (yet another) *.frf dumper. Obviously, it wont decrypt the flash payloads but it's an easy way to extract the odx and/or peek into the odx structure without opening it in an editor.

https://github.com/trick77/frfdumper

If someone has info on how to decode compression method 11 please feel free to contact me.


Title: Re: FRF and SGO - Differences?
Post by: birchbark506 on July 09, 2019, 04:41:27 PM
cmd did a update to allow write on med17 tropt7+ works with out a problem. i also got pcmflash now as well.


Title: Re: FRF and SGO - Differences?
Post by: SB_GLI on July 10, 2019, 07:24:41 AM
If someone has info on how to decode compression method 11 please feel free to contact me.

Repeating bytes are compressed into a single byte and then created as a block.  Non repeating bytes are created as a block.  There's a few header bytes before each block that defines the size of the block.  After this simple compression, each byte is XOR encrypted using a key that is unique to the type of ecu.   I am quite sure there are examples of this on this forum, but in a nutshell, that's how it works.  - this is bosch 0x11 compression/encryption.


Title: Re: FRF and SGO - Differences?
Post by: nyet on July 10, 2019, 08:48:00 AM
Repeating bytes are compressed into a single byte and then created as a block.  Non repeating bytes are created as a block.  There's a few header bytes before each block that defines the size of the block.  After this simple compression, each byte is XOR encrypted using a key that is unique to the type of ecu.   I am quite sure there are examples of this on this forum, but in a nutshell, that's how it works.  - this is bosch 0x11 compression/encryption.

Hilariously, I believe bosch stole it from Nintendo (even though there are many examples of run length compression that aren't Nintendo's)


Title: Re: FRF and SGO - Differences?
Post by: dexterash on July 11, 2019, 04:31:07 AM
Repeating bytes are compressed into a single byte and then created as a block.  Non repeating bytes are created as a block.  There's a few header bytes before each block that defines the size of the block.  After this simple compression, each byte is XOR encrypted using a key that is unique to the type of ecu.   I am quite sure there are examples of this on this forum, but in a nutshell, that's how it works.  - this is bosch 0x11 compression/encryption.
Hello

So you are saying the key is only one byte long? Or multiple bytes?
Does this apply to all ODX files with <ENCRYPT-COMPRESS-METHOD TYPE="A_BYTEFIELD">11</ENCRYPT-COMPRESS-METHOD> and, maybe, <ENCRYPT-COMPRESS-METHOD TYPE="A_BYTEFIELD">01</ENCRYPT-COMPRESS-METHOD>?

Thanks!

L.E.
I suppose you are actually talking about keys like "BiWbBuD101"...


Title: Re: FRF and SGO - Differences?
Post by: SB_GLI on July 11, 2019, 11:55:40 AM
I suppose you are actually talking about keys like "BiWbBuD101"...

ME7 = "GEHEIM"
MED9 = "CodeRobert"
MED17 = "BiWbBuD101"

:)

Interesting tidbit about nintendo, Nyet


Title: Re: FRF and SGO - Differences?
Post by: dexterash on July 11, 2019, 01:27:48 PM
I suppose these are used to decode the <data> section and apply only to BOSCH, right? I think it's only for those with 0x11 algo. Do you know anything about 0x01?

[nintendo tidbit: interesting; it's funny how the industry (re)uses different thingies]


Title: Re: FRF and SGO - Differences?
Post by: gremlin on July 11, 2019, 02:07:30 PM
Do you know anything about 0x01?

01 use Simos-Siemens and Magnetti Marelli ECUs
Crypt algo is different.


Title: Re: FRF and SGO - Differences?
Post by: gremlin on July 11, 2019, 02:34:29 PM
Next levels of compress/crypt algo from Bosch -> A1, AA and 2A use LZRB and AES technique.
Below is screenshots from my FRF coder/decoder.





Title: Re: FRF and SGO - Differences?
Post by: d3irb on July 11, 2019, 10:25:11 PM
Simos18 files use ENCRYPT-COMPRESS-METHOD=AA. The Python code I linked and posted earlier in the thread is the full implementation of method AA for Simos18. Within my implementation I also implement LZSS which is the algorithm nyet was describing. Sure enough I cribbed the decompression code from an open source Nintendo sprite decompressor, although the dictionary size (and therefore the number of bits used for the offset into the dictionary) is different between the two algorithms. LZSS is so common that I'm not sure they copied the Nintendo implementation but it wouldn't be surprising.


Title: Re: FRF and SGO - Differences?
Post by: dexterash on July 12, 2019, 12:17:04 AM
01 use Simos-Siemens and Magnetti Marelli ECUs
Crypt algo is different.

Hmmm... That means the values are different depending on the ECU type - ABS, Multimedia, Cluster etc. I think I also saw 0x01 on Continental's products.


Title: Re: FRF and SGO - Differences?
Post by: SB_GLI on July 12, 2019, 06:00:04 AM
Hmmm... That means the values are different depending on the ECU type - ABS, Multimedia, Cluster etc. I think I also saw 0x01 on Continental's products.

0x01 is just bosch encryption without compression

The 4 high bits is compression type, the 4 low bits is encryption type


Title: Re: FRF and SGO - Differences?
Post by: dexterash on July 12, 2019, 08:44:56 AM
Do you think it is the same, even if the manufacturer is not BOSCH?


Title: Re: FRF and SGO - Differences?
Post by: SB_GLI on July 12, 2019, 08:58:08 AM
Do you think it is the same, even if the manufacturer is not BOSCH?

I can provide this, though not 100% sure of it's accuracy as I have never worked with the other manufactures.

Code:
 //high nibble of data format
        public enum CompressionType : byte
        {
            Uncompressed = 0x00,
            Bosch = 0x10,
            Hitachi = 0x20,
            Marelli = 0x30,
            Lucas = 0x40,
            BoschUncompressed = 0x90 //for testing
        }

        //low nibble of data format
        public enum EncryptionType : byte
        {
            Unencrypted = 0x00,
            Bosch = 0x01,
            Hitachi = 0x02,
            Marelli = 0x03,
            Lucas = 0x04
        }


Title: Re: FRF and SGO - Differences?
Post by: dexterash on July 12, 2019, 09:32:08 AM
That's what I was "afraid". And my assumption is that's ECU-type specific.


Title: Re: FRF and SGO - Differences?
Post by: gremlin on July 12, 2019, 11:28:20 AM
That's what I was "afraid". And my assumption is that's ECU-type specific.

This compress/crypt byte haven't exact meaning as absolute value.
It's  depends from manufacturer, control unit, etc
Some examples on the picture. Of cause it isn't all possible variants.


Title: Re: FRF and SGO - Differences?
Post by: dexterash on July 12, 2019, 01:05:10 PM
Thanks. Saw your earlier posts too. From my point of view, this is kind of tricky/non-standard. Or maybe the algos are the same across different manufacturers? And let's say a key or a constant/fixed value changes?


Title: Re: FRF and SGO - Differences?
Post by: gremlin on July 12, 2019, 02:18:43 PM
From my point of view, this is kind of tricky/non-standard.

No. It's fully standard action.
Open ISO14229, or 14230 or any other automobile standard, described down/up data transfer/
Read about DFI (data format identifier) parameter and you will see:

dataFormatIdentifier
This data parameter is a one-byte value with each nibble encoded separately. The high nibble specifies the
“compressionMethod” and the low nibble specifies the “encryptingMethod”. The value 00 hex specifies that no
compressionMethod nor encryptingMethod is used. Values other than 00 hex are vehicle-manufacturer-specific.


The manufacturer can assign DFI value as they like without violating the requirements of the standard.




Title: Re: FRF and SGO - Differences?
Post by: dexterash on July 12, 2019, 02:41:11 PM
Thanks for that lesson - didn't knew that. Although, sometimes, I saw or stumbled upon implementations that are not documented/nor standard. Especially at lower levels, debug mode, security access & co. But, yet again, my experience is scarce about this.

Do you think that, for example, Japanese, Chinese or Korean cars would follow the same rule?


Title: Re: FRF and SGO - Differences?
Post by: dragon187 on May 25, 2020, 01:03:04 PM
Hi
Are there any news on this?

If anyone have built some software please contact me.

I would buy.

Many thanks

BR
 ;)


Title: Re: FRF and SGO - Differences?
Post by: Sagishm on October 11, 2020, 03:33:06 PM
Hi All,
i try to understand what is the way to convert file with `ENCRYPT-COMPRESS-METHOD  = 11` but is not BCB like in med17.
i had success to convert files from med17 but not from dsg box.

someone can give me more information?

example to file is attached.


Title: Re: FRF and SGO - Differences?
Post by: gremlin on October 12, 2020, 07:54:36 AM
someone can give me more information?
example to file is attached.
This TCU use 256 bytes encryption table and LZZ-compression.


Title: Re: FRF and SGO - Differences?
Post by: Sagishm on October 13, 2020, 03:12:02 AM
This TCU use 256 bytes encryption table and LZZ-compression.

256 bytes encryption = AES256?
Where can i find the key and iv?

thanks


Title: Re: FRF and SGO - Differences?
Post by: gremlin on October 13, 2020, 10:12:24 AM
256 bytes encryption = AES256?

Nо, it's not AES.
In-house manufacturer's algorithm.


Title: Re: FRF and SGO - Differences?
Post by: Sagishm on October 13, 2020, 11:21:54 AM
Nо, it's not AES.
In-house manufacturer's algorithm.
So basically to find the algorithm i need to dissembling bin file and that’s will be in the bootloader?


Title: Re: FRF and SGO - Differences?
Post by: vagabond45 on November 25, 2020, 04:08:42 PM
-


Title: Re: FRF and SGO - Differences?
Post by: IamwhoIam on November 27, 2020, 06:01:40 AM
Hi. Trying to find some info on the compression algo used on the file attached. med17.85 ecu..

You can dm me as well if this is too off topic to the rest of the convo.

It wouldn't hurt if you provided some more info about what the file comes out of, just sayin'


Title: Re: FRF and SGO - Differences?
Post by: vagabond45 on December 16, 2020, 12:21:48 AM
It wouldn't hurt if you provided some more info about what the file comes out of, just sayin'

My fault. Updated the initial post. The file is an eeprom read i took from ktag.


Title: Re: FRF and SGO - Differences?
Post by: navatar_ on December 26, 2020, 09:50:17 PM
This TCU use 256 bytes encryption table and LZZ-compression.

Can anyone shed some light on the 'LZZ' compression algo? At first I assumed it was synonymous with LZSS but unless I am missing something obvious, the compressed bitstream looks quite different to that of the LZSS used in other ECUs like Sim18.


Title: Re: FRF and SGO - Differences?
Post by: Wommesz on December 27, 2020, 10:51:36 AM
Quote
Can anyone shed some light on the 'LZZ' compression algo? At first I assumed it was synonymous with LZSS but unless I am missing something obvious, the compressed bitstream looks quite different to that of the LZSS used in other ECUs like Sim18.

This is far from an answer, but maybe it's helpfull to know:

We might be looking at something similar, even though this is from a Simos 10 ECU:
http://nefariousmotorsports.com/forum/index.php?topic=18832.msg142207#msg142207
I haven't gotten any further since that post. Maybe with a decrypted boot section and a dissasembler it's possible to locate the decryption routine.


Title: Re: FRF and SGO - Differences?
Post by: moodz on January 09, 2021, 11:14:39 AM
not sure if this helps here

Search for Patent Application 20150333766

A method for run time zero byte compression of data on a communication bus of a vehicle includes determining a number of zero bytes provided in a data frame. When there are enough zero bytes, an encoding byte is generated that maps the locations of the zero bytes in the data frame. A data length code related to the number of non-zero data bytes and the encoding byte is provided in a device header. The data length code has a value less than an uncompressed data frame. The compressed data frame is transmitted with the encoding byte and the uncompressed non-zero data bytes. To decompress the compressed data frame, the encoding byte maps the locations of the zero bytes for a data frame. The non-zero data bytes are then provided at the proper locations to recreate the data frame.


Title: Re: FRF and SGO - Differences?
Post by: nyet on January 10, 2021, 02:03:32 AM
Jesus I hate patent authors.

"the encoding byte maps the locations of the zero bytes for a data frame" is mentioned twice.

But zero mention of how the locations are encoded in the encoding byte, nor does it mention if the length of the consecutive zeros are encoded, though it is obvious that the encoding would be pointless without it.

They really are scum. Describe a method just enough to make the description utterly useless so the patent cannot be used to reproduce the technology, which (ostensibly) is the purpose of a public patent system (not to enrich the authors, even though that is the common wisdom of the purpose of a patent system).


Title: Re: FRF and SGO - Differences?
Post by: navatar_ on March 09, 2021, 01:59:18 PM
Next levels of compress/crypt algo from Bosch -> A1, AA and 2A use LZRB and AES technique.

Gremlin what exactly is LZRB? I know of LZRW & LZJB but I can't find any reference to an LZRB compression algo outside of this thread?


Title: Re: FRF and SGO - Differences?
Post by: gremlin on March 09, 2021, 05:36:01 PM
I can't find any reference to an LZRB compression algo outside of this thread?

LZRB means manufacturer specific (Robert Bosch) modification of LZ compression technique.


Title: Re: FRF and SGO - Differences?
Post by: navatar_ on March 09, 2021, 08:59:21 PM
LZRB means manufacturer specific (Robert Bosch) modification of LZ compression technique.


I see. Thanks gremlin.


Title: Re: FRF and SGO - Differences?
Post by: d3irb on August 25, 2021, 03:48:34 PM
This TCU use 256 bytes encryption table and LZZ-compression.

Here is the encryption algorithm for compression/encryption "0x11" for DQ250-MQB:

https://github.com/bri3d/VW_Flash/blob/master/lib/crypto/dsg.py

This one is interesting because it isn't a real accepted crypto algorithm (like AES) or an XOR keystream algorithm (like older Bosch) - it's a progressive substitution cipher.

Here's the substitution key data:

https://github.com/bri3d/VW_Flash/blob/master/data/mqb_dsg_key.bin

The compression algorithm is LZSS so my existing decompressor seems to work - I am not sure about the comments made elsewhere in this thread about the data stream looking different from Simos/"Audi" LZSS, maybe someone was looking at the encrypted data instead of the compressed data?

https://github.com/bri3d/VW_Flash/blob/master/extractodx.py now has a `--dsg` flag to extract MQB DSG ODXes - I tested on a few and it produced good looking binaries.

I think the algorithm is similar for some other DSG models but with different 256-byte cypher data.

To figure this out wasn't so bad, I downloaded a DSG bench read from this very thread and loaded it up in Ghidra (it's Tricore, so 0x80000000 base address). The DSG's UDS handler is a simple switch construct so it was pretty easy to find, and then inside of the 0x36 TransferData handler there's a call to a routine that both decrypts and decompresses a block, with a simple xref to the key data. I've attached a screenshot of the decryption method as Ghidra pseudocode for the curious, although the Python implementation linked above is probably easier to read.


Title: Re: FRF and SGO - Differences?
Post by: prj on August 26, 2021, 04:43:19 AM
Since Python seems to be the name of the game, and we're posting old stuff.
Here's something I made a long time ago.

This just breaks all of the XOR faux security on the fly. As long as it's BCB + XOR encrypted, you don't need the key.

Remember to XOR by 0xFF first, before applying this algorithm, if you're dealing with SGO.

Code:
def deleterepeat(s):
s = binascii.hexlify(s)
i = (s+s).find(s, 1, -1)
return binascii.unhexlify(s) if i == -1 else binascii.unhexlify(s[:i])

def stripBCBHead(imgxor):
return imgxor[imgxor.index(b"\x1A\x01") + 2:]

def freqtable(data, klen):
freqtable = {}
for x in range (0, klen):
curfreq = {}
for y in range (0, 256):
curfreq[y] = 0
freqtable[x] = curfreq

for nr in range(len(data)):
freqtable[nr % klen][data[nr]] += 1

return freqtable

def findXORkeyfreq(bcbdata, byte, confidence, maxlen):
data = stripBCBHead(bcbdata)
keyFound = False
key = bytearray()
debug = config.getint("main", "debug", fallback=0)

for curlen in reversed(range(4, maxlen+1)):
fqtable = freqtable(data, curlen)
key = bytearray(curlen)

avgconf1 = 0
for fqpos in sorted(fqtable):
sortedlist = sorted(fqtable[fqpos].items(), key=lambda x: x[1], reverse=True)
avgconf1 += 100 - sortedlist[1][1]/sortedlist[0][1]*100
key[fqpos] = sortedlist[0][0] ^ byte

avgconf1 = avgconf1/curlen

if (avgconf1 >= confidence):
key = deleterepeat(key)
return key
return b""

Call like:
findXORkeyfreq(bcbdata, 0xFF, 50, 32)

Recommend calling it on the biggest section or just foreach all sections until a key is found, etc...


Title: Re: FRF and SGO - Differences?
Post by: gremlin on August 26, 2021, 10:08:08 PM
I think the algorithm is similar for some other DSG models but with different 256-byte cypher data.

You are right.
At least 20 variants of 256 encrypted data tables are used in DSG TCU control units.


Title: Re: FRF and SGO - Differences?
Post by: dstar on December 13, 2021, 05:34:39 AM
Hello.
But what about other modules? For example I have one A6 dashboard also with encryption 11. It is Bosch with Fujitsu MCU, in the ODX there are RSA keys and data sectors:

              <SHORT-NAME>FD_1DATA</SHORT-NAME>
              <LONG-NAME>1 DATA</LONG-NAME>
              <DATAFORMAT SELECTION="BINARY"/>
              <ENCRYPT-COMPRESS-METHOD TYPE="A_BYTEFIELD">11</ENCRYPT-COMPRESS-METHOD>

Does any one knows the encryption algorithm 11 for Fujitsu?
Thanks.