An error in LibreOffice corrupts password protection in xls-files of Microsoft Excel
Online service AccessBack.com uses the Passcovery original rainbow tables to find an encryption key (only 40-bit keys!), then it decrypts a password-protected Microsoft Excel / Word 97-2003 document and removes the Password to Open
The success rate is so close to absolute that we consider it 100%.
To deal with other individual cases our programs offer the encryption key search option. Though it takes more time it ensures that the key gets found with a 100% guarantee.
As soon as neither rainbow tables nor brute force search worked out and failed to find the encryption key we decided to overhaul the first strange file we got hold of.
It turned out that the storage of encryption key data was empty and the data per se were not encrypted, however password protection indicator was toggled on. Also right there in the Microsoft Excel file there was a record pointing to the likely culprit: Calc…
Later on the owner of one of these strange files clarified the situation:
Some of those who uploaded their files for decryption at AccessBack.com share the same story – they created and protected their files in Microsoft Excel but as soon as they edit them in LibreOffice something goes wrong and the passwords stop working.
Removal of «password protection» from Excel files corrupted in LibreOffice
Open the file in a HEX editor and find the header of the data stream. It starts with the print 0x09081000 and is usually located at 0x600, however variations are possible (the beginning of the header is always aligned to 0x200).
Go to the encryption header. It starts with the print 0x2F003600.
The beginning of data stream and encryption header in Microsoft Excel
See if the file you opened is corrupted in the following way – in the data block behind the printout of the encryption header instead of a “dense” set of random numbers there are zeros and the block ends with a readable “Calc”. That’s what the problem looks like:
Typical appearance of encryption header of an Excel file corrupted in LibreOffice
By contrast, here's what this data block looks like in an encrypted and uncorrupted Excel file:
Typical appearance of encryption header in an encrypted Excel file
Change the value in the header print from 0x2F to 0x22 to get 0x22003600:
Disable the use of encryption header in the corrupted file
Zeros in the encryption header indicate that the data needed to decrypt the file are not specified. That means that actually the file and the data stored in it are not encrypted but password protection indicator is toggled on.
Due to some internal error LibreOffice saved the password-protected file without protection but at the same time left a record indicating that protection is enabled.
And you have removed that record by just editing one byte.
Save the changes in HEX editor and open the file in Microsoft Office or LibreOffice. You no longer need the password for this. There is no protection in the file and no password is set.