How to force binary hash algorithm in Ensemble production?
TL;DR: If I set an Ensemble Production Service based on EnsLib.File.PassthroughService to a Binary charset encoding, it breaks the file handling. Any ideas?
Full long post:
All,
I set up an Ensemble production to transfer files via SFTP, which works fine sending the files to my Linux server. Then I was informed that we needed to save a hash of the file, for comparison with the destination to verify the file arrived unmodified. I looked at the base64 encoded hash on my linux server:
G7QWAP6FcLInFWP8ECRL/EI2NfKdaf6TtrpwYuvSOEc=
and I created a class to extend the StreamContainer to add a hash value:
{ Property Hash As %String(MAXLEN = 256); }
which seems to work fine. But when the production runs, the hash is listed as:
Vn0ygZZN8DewD8KfKKYF8BTV9K1LsuFBEXnAkqXBEFs=
I narrowed that down to Ensemble automatically choosing the stream type - in this case, it sees it as a FC type stream - File Character - but Linux sees it as a Binary encoding. I verified the issue with this bit of code:
New bstream,cstream,sc,bhash,chash,zfile
Set zfile="d:\Patch\Test_File_1.CSV"
Set bstream=##class(%Stream.FileBinary).%New()
Set sc=bstream.LinkToFile(zfile)
Do bstream.Rewind()
Set bhash=$SYSTEM.Encryption.SHAHashStream(256,bstream)
W "Binary: "_$SYSTEM.Encryption.Base64Encode(bhash),!
Do bstream.%Close()
Set cstream=##class(%Stream.FileCharacter).%New()
Set sc=cstream.LinkToFile(zfile)
Do cstream.Rewind()
Set chash=$SYSTEM.Encryption.SHAHashStream(256,cstream)
W "Char: "_$SYSTEM.Encryption.Base64Encode(chash),!!
Do cstream.%Close()
Q
And when run, here's the output:
ENSDEMO>D ^HASHCHECK Binary: G7QWAP6FcLInFWP8ECRL/EI2NfKdaf6TtrpwYuvSOEc= Char: Vn0ygZZN8DewD8KfKKYF8BTV9K1LsuFBEXnAkqXBEFs=
I found a setting in the EnsLib.File.PassthroughService to manually set the charset to Binary, and when I set that, the production will give me the correct hash and sends the file - but then recreates the file with a randomized OriginalFilename, waits 5 seconds (or whatever's set in the Retry Interval parameter), and sends it again... and again... and again.
When I look at the message viewer, the <Stream></Stream> is empty:
<?xml version="1.0" ?> <!-- type: SFTP.HashFile id: 281 --> <HashFile> <OriginalFilename>C:\Export\aPvlvsYdsoNpng.CSV </OriginalFilename> <Stream></Stream> <Type>FB </Type> <Attributes> <AttributesItem AttributesKey="1" xsi:nil="true"></AttributesItem> </Attributes> <Hash>G7QWAP6FcLInFWP8ECRL/EI2NfKdaf6TtrpwYuvSOEc= </Hash> </HashFile>
Even though file is sent intact successfully. (You can see an example of the 'random' filename above.)
Any ideas that would be easier than rewriting a custom EnsLib.File.PassthroughService and/or Ens.StreamContainer to only handle binary encoded streams?
Thanks for any and all input!
Try change TranslateTable:
Vitaliy,
Thanks for the code, but how would I integrate changing the the TranslateTable in the DTL of my Production Process? That's where the hash is getting created; I tried several permutations of '
set source.Stream.TranslateTable=""
' but all gave me a <PROPERTY DOES NOT EXIST> error. The CharEncodingTable is a Readonly Internal value, and trying to changing that gives me a <CANNOT SET THIS PROPERTY> error.Although, looking at the source code of the Ens.StreamContainer class, that might give me an idea how to rewrite the class _easily_ to accomplish what I need...
Thanks for the input!
Yes, recreates in the same folder, as the production then re-sends the file (with the randomized filename) via SFTP to the remote server.
If it makes a difference, the production is on a Windows 10 system, although I have access to a Linux server for testing as well.
Settings:
File Path: C:\Export\
Archive Path: C:\Export\CSV_Complete
[[ I have two different business services looking for two different extensions)
Work Path: [[ Null ]]
I don't see a 'DeleteFromServer' setting... will continue digging.
Also, Conform Complete: Readable (default)
File Access Timeout: 2
Hope this helps, and thanks!
You can solve the issue "head-on", namely, to re-convert character stream using %IO.StringStream:
Result:
SubdirectoryLevels: 0
I honestly didn't think that it would make a difference, as when Charset = Native, it works fine. Only when Charset = Binary is when it acts odd.
That said, I changed ArchivePath to c:\Export2\CSV_Complete\ -- restarted the production and tested -- no change in behaviour.
Thanks!
I'm out of ideas. There should not be any changes between file and character streams besides encoding.
I think you need to share a minimal sample that reproduces this error or contact the WRC.
What's your SubdirectoryLevels setting value?
Try to move Archive Path outside of File Path.
Recreates in the same folder?
What is the value of FilePath, WorkPath, ArchivePath, DeleteFromServer settings?
You need to solve the problem with file recreation, as specifying "binary" as a Charset setting gives you the correct hash.